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FOREWORD 



In recent years, short-term skill training in postsecondary institutions, as well as in business 
and industry, has expanded rapidly. Technological change and structural shifts in the economic 
base have created critical skill shortages in many areas. Government employment and training 
programs have also increasingly emphasized short-term training as a way of preparing the unem- 
ployed and economically disadvantaged for entry into the job market. 

Despite the extent of short-term training, planners, administrators, and evaluators of short- 
term programs have had few practical guidelines to assist them in selecting, designing, and 
implementing their programs. This guide responds to their needs by presenting an overview of 
short-term training. An introduction to evaluation models, designs, and criteria is provided and 
specific approaches to evaluating short-term programs are discussed. 

The Office of Vocational and Adult Education, U.S. Department of Education, provided the 
sponsorship for this report, which was developed by the Evaluation and Policy Division of the 
National Center for Research in Vocational Education under the leadership of N. L. McCaslin, 
Associate Director. We wish to thank project staff— Frank C. Pratzner, Project Director; Elizabeth 
V Dubravcic, Program Associate; Christian A. Chinien, Graduate Research Associate, and Alan 
Kohan, Graduate Research Associate— for their work in preparing the report. 

Ann Blalock, Employment Security Department, Seattle, Washington; Oliver Cummings, 
Arthur Anderson and Company, St. Charles, Illinois; Winifred DeLoyaza, Interorganizational Rela- 
tionships, Ltd , Altament, New York; and Russell Paulsen, North Central Technical Institute, 
Wausau, Wisconsin, served as members of the project's panel of consultants. 

We would also like to extend our appreciation to Charles Branch and Robert Newton of Tri- 
dent Technical College, Charleston, South Carolina; Mitzi Ellis and Jean Stuckert of the Crawford, 
Hancock, Marion, and Wyandot Counties Private Industry Council, Inc. in Ohio; Nelson RonsvaMe, 
State Bureau of Economic Development, Albany, New 'ork; David Mathes, Finger Lakes Regional 
Education Center for Economic Development, Mount Morris, New York; Kay Ford, Monroe Com- 
munity College, Rochester, New York; Kay Grastie, Marth* Herbert, and Jerry Sams of Greenville 
Technical College, Greenville, South Carolina; Marshall Goldberg UAW-Ford National and 
Development Training Center, Dearborn, Michigan; James Rossbach, Shirley Johnson, and Jim 
Swanson of Hennepin Technical Centers, Plymouth, Minnesota; Bill Warner, Chuck DeVore, Jeff 
Larson, and Bob Tuchel of 916 Area Vocational Technical Institute, White Bear Lake, Minnesota, 
and Dale Tippett of Columbus Technical Institute, Columbus, Ohio. These individuals and their 
staffs coordinated and hosted site visits and provided invaluable information in interviews. 

We were fortunate to have benefited from the critical reviews and recommendations of 
Marshall Goldberg, UAW-Ford National Development and Training Center, Dearborn, Michigan, 
and Paul Zachos, Bureau of Occupational Research, Albany, New York. Roy Butler and Floyd 
McKinney, Senior Research Specialists of the National Center, also reviewed the publication and 
we are grateful for their contributions. Although we do not mention them by name, we thank the 
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many individuals in State departments of vocational education, postsecondary institutions, and the 
National Center who provided us with resources, references, and information regarding short-term 
skill training programs and their evaluation 

Finally, the information in this document would not have been "accessible" without the patient 
help of Cathy Jones, Lou Pierson, and Jeannette Painter in manuscript preparation The editorial 
assistance of Janet Kiplinger and Jeanne Desy is especially appreciated 



Robert E Taylor 
Executive Director 
The National Center for Research 
in Vocational Education 
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EXECUTIVE SUMMARY 



Short-term skill training programs have developed and grown rapidly in response to cutical 
skill shortages and worker displacements caused by technological advancement, the trend toward 
an information- and service-based economy, and increasing foreign competition in the production 
of goods. Whereas the amount of public and private resources devoted to these training programs 
has soared, their evaluation has received considerably less emphasis. This publication was devel- 
oped tc provide local instructors, administrators, and evaluators of short-term training programs 
with information and insight to assist them in evaluating these programs. Although this publication 
is designed primarily for use by those ndividuals responsible for developing, designing, coordinat- 
ing, and assessing short-term training programs in postsecondary institutions, providers of train- 
ing m industry- or government-sponsored agencies may also find this guide useful 

Short-term programs share several characteristics. The most obvious, perhaps, is their length 
Most are a year or less in duration. Short-term programs are defined here as lasting nine months 
or less, to distinguish them from 1-year certificate programs. A second characteristic of short-term 
programs is their emphasis on training— providing specific, narrowly defined job-related learning 
experiences— rather than education. 

Providers of short-term training vary. They include employers, labor unions, trade associa- 
tions, private and public secondary institutions, government sponsored employment and training 
agencies, and the military. Short-term training is directed at meeting the immediate and unique 
labor force needs of a particular community, employer, and/or groups of individuals The clientele, 
or trainees, are frequently sponsored either by their t • 'oyer or through public training funds 

Instructors of short-term skill training generally teach on the basis of part-time temporary con- 
tracts. Many are recruited from the ranks of existing staff at local postsecondary institutions, 
others are recruited from business and industry Instructors of short-term training programs must 
be knowledgeable and skilled in instructional strategies and content, and adept in modifying and 
fitting training to client needs. 

In many short-term programs curriculum is systematically designed to produce specific job 
and task-related competencies, since program outcomes and cost effectiveness are important 
Employers are often directly involved with program design and planning 

Program evaluation— a systematic process of obtaining and providing information about the 
worth or merit of a program for purposes of decision making— nan be as useful for short-term pro- 
grams as for longer programs. Evaluation models include: systems analysis, behpvioral objectives, 
decision making, goal-free evaluation, art criticism, accreditation, and transac* jnal models The 
usefulness for short-term programs of the systems analysis model lies in its emphasis on educa- 
tional efficiency. The behavioral objectives model stresses objectivity in identifying and measuring 
behavioral outcomes The decision-making model emphasizes comprehensive review and analysis 
of the content, inputs, process, and products of the program for formative input into program deci- 
sions. Goal-free evaluation is especially noted for its assessment of both intended and unintended 
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outcomes The art criticism model is useful for improving program standards The accreditation 
model emphasizes the review of multiple programs according to predetermined criteria Finally, 
the transactional model provides qualitative information for judging programs on the basis of abso- 
lute and relative standards. 

Evaluation design may be fixed— systematically planned in id"ance— or emergent—changing 
in response to changing program activities, programs, and audiences. It may be tailored to meet 
the needs of formative or summative evaluation. It may be experimental or quasi-experimental, 
focusing on differences between outcomes for control and experimental groups, or it may be 
naturalistic and unobtrusive, using intact groups in natural settings, thereby eliminating some of 
the ethical and practical problems that arise in experimental evaluation. 

If the evaluation is to be useful, it must provide sufficient information for decision making, and 
it must be presented clearly and in a timely fashion. If it is to be feasible, the evaluation procedures 
should be practical, politically viable, and cost-effective. The rights of human subjects, conflicts of 
interest, and the need for balanced reporting should be considered in designing, conducting, and 
presenting evaluation results. 

Finally, the accuracy of evaluation depends upon several considerations: a clearly defined 
object of evaluation, appropriate sources of information, adequate contextual information, under 
standing of evaluation purposes and procedures by those involved in the process, and reliable and 
valid collection and measurement of information. 

Evaluability assessment is t ie process of deciding whether the evaluation is likely to fulfill its 
goals and merit the expended effort. The evaluability assessment provides answers to such ques- 
tions as, What is the demand for evaluation? What is the source? Is it feasible? What decisions will 
be served by evaluation? and What part of the program development should receive what kind of 
emphasis in the evaluation? This is a vital first step in any evaluation of short-term training 
programs. 

Needs assessment is a process of both program development and program evaluation The 
needs assessment process identifies the gaps between what should be and what actually is, priori- 
tizes these gaps or needs, and selects those to be addressed by the program. In planning needs 
assessment, the evaluator must determine who will be included in the process, the strategy to be 
used, and the methods by which information will be analyzed and reported. In short-term training, 
needs assessment activities generally focus on determining individual trainee, company, or indus- 
try needs. 

Input evaluation focuses on determining the best use of the available resources to ensure that 
program goals are met. Sources of information for input evaluation might include existing program 
review guidelines for vocational education and program standards specifications. One means by 
which program design may be evaluated and validated is through the involvement of experts This 
can take the form of an advocacy team, task force, advisory committees, employer representative, 
or curriculum developer. 

Process evaluation is utilized to monitor program implementation and p r ogram operators for 
potential problems or inconsistencies between the original program design and its actual process 
This is an important tool for providers of short-term training programs, for it enables them to 
ensure that programs are on target prior to the end of training. For the majority of these programs, 
there is no chance to improve delivery once the training program is ended. 



Outcome evaluation addresses the question of whether and to what extent immediate program 
objectives have been met. Program outcomes/objectives may focus upon individual outcomes (i e . 
student satisfaction, achievement), group outcomes (percent of placement, retention rate), and 
outcomes for other clients and stakeholders (providing skilled workers to industry) Assessment of 
multiple outcome measures permits evaluators of short-term skill training programs to control for 
spurious results and to obtain a clear and more complete picture of program effectiveness and 
productivity 

Impact evaluation addresses the broader short- and long-term effects of training for the indi- 
vidual, company, or business and for society as a whole. Examr' s of the way in which each may 
be measured are higher earnings over a period of time and career development, increased produc- 
tivity and improved quality of product; reduction in unemployment and more equality of oppor- 
tunity in employment. Assessing the impact of short-ierm skill training is becoming increasingly 
relevant as the level of both public and private investment in such training soars. 

Cost-benefit analysis and cost-effectiveness analysis are tools for assessing the efficiency of 
program operation. Cost-benefit analysis may be used in the planning and design of specific pro- 
grams, whereas cost-effectiveness may be used to compare the effectiveness and the efficiency of 
two or more programs 

Aggregate program review and peer review are two promising approaches to evaluating short- 
term skill training. Aggregate review is directed at assessing a group of programs at the institu- 
tional level C haracteristics such as program costs, attendance, enrollments, expenditures and 
revenues, student achievements, and placement rates or other measures of outcomes and impact 
might be monitored for all programs and used to identify programs for in-depth review. Peer 
review involves the use of external evaluators to evaluate the program or group of programs Peers 
may be colleagues from outside the institution involved in providing similar services, or they may 
be private training consultants. Peers may fruitfully be utilized to provide formative evaluation of 
needs assessment, program design, and program implementation procedures. 

The evaluation of short-term programs is both easier and more difficult than evaluation of 
long-term programs. It is easier because the objectives of short-term programs are usually clearly 
and precisely specified in advance, and more difficult because the resources earmarked for evalua- 
tion tend to be minimal or nonexistent and because reassessment is not possible Yet evaluation is 
crucial to quality assurance Purthermore, it provides a means of systematically accounting for the 
dollars spent. 
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CHAPTER 1 



INTRODUCTION 



The number of short-term occupational skill training programs in public postsecondary insti- 
tutions and in businesses and industries is rapidly increasing. Three types of programs account for 
much of this growth: (1) those meeting critical skill shclages in particular regions, (2) those pro- 
viding customized training to particu'ar businesses and industries, and (3) those reducing un- 
employment and meeting the needs of unemployed workers (for example, dislocated workers, 
displaced homemakers, and disadvantaged youth and adults). 

Because training programs do address these diverse needs, they differ substantially in their 
purpose, design, and operation. However, they share certain characteristics that set them apart 
from longer, more established programs. In general, funding is scarce for these programs. They 
are more closely related to employment than most long-term programs. They focus more ciearly 
and directly on specific objectives. And because short-term training programs a; e a relatively 
recent phenomenon, there have been few guidelines for assessing their effectiveness. 

T*e irregular demand for short-term programs, the fact that they are short-term and one-of-a- 
kind, and the urgent need to "get the job done" have placed them outside the scope of regular, 
systematic evaluations in many postsecondary institutions. Relatively low levels of f jnding have 
discouraged many vocational educators from attempting in-depth evaluations of their programs 
Nevertheless, as more private and public resources are invested in short-term programs, the need 
fot evaluation will become more apparent. In addition, administrators must decide the extent to 
which their institutions can become engaged in these programs and must determine how teaching 
services can be improved. They also need to know what impact their short-term training programs 
are having on the trainees, the employers, and the community as a whole. 

Program evaluation in general has been extensively studied, but not much has been written 
about evaluating short-term skill training programs. The purpose of this guidebook is to provide 
vocational planners and evaluators with information that can be used in evaluating short-term 
training programs. The guidebook is intended lor use by educators in postsecondary institutions 
tha! are developing and implementing short-term programs, those providing service under the Job 
Training Partnership Act (JTPA), and employers in the private sector who provide in-house skill 
training. 



Overview of th j Research Project 

The goal of the project on which this guidebook was based was to develop guidelines for the 
evaluation of short-term skill training programs. In order to accomplish this goal, three major 
objectives were established: 

• To determine the nature and characteristics of various types of short-term skill training 
programs, with reference primarily to their purposes, design, and operation 
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• lo determine program evaluation approaches appropriate for various types of short-term 
skill training 

• To ascertain specific criteria appropriate for evaluating various types of short-term skill 
training programs primarily in terms of their operations (processes) and outcomes 
(products) 



Methodology 

A three-pronged approach vas u? ed to accomplish the project objectives* 

• Reviews and syntheses of the literature and research relevant to each of the three specific 
objectives were conducted. These reviews focused primarily on the purpose, design, and 
operation of short-term skill training programs and the effectiveness of such programs 

• Site visits, interviews, and observations were conducted at a number of postsecondary 
institutes and other sites that operate various types of short-term skill training programs 
The purpose of these visits was to sharpen and refine the descriptions of such programs 
and of useful program evaluation approaches and to define the specific criteria noeded to 
evaluate such programs. 

• Finally, a panel of consultants— experts in vocational evaluation— was convened to 
examine the program characteristics, to help identify the most appropriate evaluation 
approaches for different types of programs, and to help identify specific evaluation 
criteria. 



Literature Review 

The initial project activity was the review and synthesis of relevant literature and research. 
Computer searches were conducted to identify research studies and related reports pertinent to 
the project's three major objectives; Educational Resources Information Center (ERIC) and the 
National Technical Inforn ition System (NTIS) databases were used. The key ERIC descriptors 
chosen focused on shorHerm training programs, program evaluation, and specific evaluative 
criteria. Key documents pertaining to evaluation in both education *nd training were identified 
After reviewing the key documents and abstracts of the relevant studies, project staff synthesized 
imbings regarding the nature and characteristics of short-term programs, program evaluation 
approaches, and specific evaluation criteria. 



Site Visits 

In order to observe short-term training programs firsthand, site visits were conducted The 
sites visited are listed in appendix A. Sites were selected to provide diversity in location, institu- 
tional size and type, and kind of short-term training offered in terms of program purpose, design, 
and operating procedures. 

National Center staff interviewed program directors and evaluutors. Programs in operation 
were observed. The emphasis of the site visits was to observe key similarities and differences 
among programs, to explore the nature of evaluation in these programs, and to determine the 
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evaluation criteria being utilized The information collected at these site visits provided insight into 
ways the major objectives of this project could be achieved Specific examples of the lessons 
learned and descriptions of specific ways to operate an evaluation program that were observed are 
provided throughout the guidebook 



Panel o! Consultants 

A panel of consultants was assembled to examine and elaboraie upon the program character- 
istics, evaluation approaches, and evaluation criteria identified by National Center staff as a result 
of the literature review and site visits. Four consultants were convened for a 1-day panel discus- 
sion The four panel members (see appendix B) were selected to represent the following roles, 
responsibilities, and organizations: 

• One practitioner from a community or te :hnical college who develops short-term and 
other kinds of training programs for specific industries and employers 

• One State-level person with responsibility for Statewide evaluation of programs serving 
business and industry and of those serving special populations provided with specialized 
training programs 

• One human resource development person from industry who develops and administers 
company training programs 



How to Use This Guidebook 

This guidebook is designed to describe the many possibilities in evaluating short-term pro- 
grams. It is written for the trainer or administrator whose experience in evaluation is limited, the 
early chapters present an overview of evaluation theory and of the commonly used evaluation 
models Many vocational educators and incustry trainers have made only limited use of evaluation 
in the past and may wish to have access to this information. 

Chapter 2, "A Look at Short-Term Training Programs," compares the general characteristics 
of short-term and long-term training programs in terms of program duration, philosophy, organi- 
zational context, purpose, clientele, instructional staff, funding, curriculum, and linkage with em- 
ployers. This chapter may be especially useful to those who are developing new programs. 

Chapter 3, "Program Evaluation: An Overview," briefly reviews evaluation models that might 
be useful in addressing evaluation needs of short-term skill training programs. Various ways to 
design an evaluation are presented, as well as some criteria for conducting evaluations This chap- 
ter also offers an evaluation framework that incorporates evaluat ve criteria appropriate for short- 
term programs. 

Chapter 4, "Options in Evaluation Design," discusses key administrative concerns in planning 
and impleme y\ing evaluation. This chapter, based upon the insights gained during site visits and 
recommendations from panel members, presents guidelines for conducting all types of evalua- 
tion— evaluabilty assessment, needs assessment, input evaluation, process evaluation, outcome 
evaluation, impact evaluation, and cost-benefit/cost-effectiveness analysis. 
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Chapter 5, "Choosing an Evaluation Approach," presents decision makers with some addi- 
tional guidelines in selecting an approach to evaluating short-term skill training programs Two 
innovative approaches to evaluation of short-term programs are presented These techniques are 
in use in other kinds of programs, but their use in evaluating short-term skills training is innovative 
and holds real potential for these programs Finally, the chapter discusses how several evaluation 
component can be effectively combined and utilized in assessing even the shortes* of training 
programs. 



CHAPTER 2 



A LOOK AT SHORT-TERM TRAINING PROGRAMS 



A good way to understand the nature of short-term training programs is to compare them 
with l^ng-term programs Chapter 2 does this by comparing the following elements (1) duration, 
(2) philosophy, (3) organizational context, (4) purpose, (5) clientele, (6) instructional staff, (7) cur- 
riculum, and (8) linkage with employers. Admittedly, these categories may not be specific enough 
for fine d*stinctions, but they do form a useful framework for asce* aining the fundamental nature 
of the short-term program. 



Duration 

Paulsen (1981) defines short-term skill training programs as "usually a year or less in duration 
designed to train, retrain, or upgrade the skMIs of workers" (p. 1). Warmbrod and Faddis (1983) add 
that short-term draining has been an important approach to meeting upgrading or retraining needs 
that do not require the completion of an associate degree or 1-year certificate. Such training may 
involve accelerated courses, short courses, workshops, seminars, or lectures (p. 25). 

Long-term training programs generally last 1 year or longer. Typically, the shortest are 1-year 
certificate or diploma programs. They may include concentrations in robotics, laser optics, com- 
puter electronics, hazardous waste control, and training activities that support economic devel- 
opment. For the purposes of this guidebook, short-term skill training programs are somewhat 
arbitrarily seen as those lasting 9 months or less. This seems to distinguish them clearly from cer- 
tificate programs of approximately 1 year in duration. In practice, however, it appears that the 
majority of short-term programs are less than 6 months in duration. 



Philosophy 

The underlying philosophy and rationale for much of short- and long-term training he in their 
impact on maximizing productivity and efficiency in the workplace. However, the way in which 
short- and long-term programs propose to deal with the broader questions of increasing productiv- 
ity in the workplace differ in principle. Briefly stated, short-term programs tend to focus on train- 
ing, whereas long-term programs tend to focus on education. Although some might argue that 
these terms are interchangeable, a look at their definitions indicates that they are different con- 
cepts pertaining to different learning experiences. 

Moss (1983) offers this analysis: 

Webster's Third New International Dictionary, Unabridged (1976) and the Dictionary of 
Education, 3rd edition, (1973) describe education as a process of rearing, bringing up, 
developing, and fostering growth and expansion of knowledge, and other desirable quali- 
ties of mind and character that will be of positive value to the individual in the society in 



which she or he lives Training, on the other hand, is seen more as shaping or develop- 
ing individuals, through drill and discipline, to attain clearly determined, readily demon- 
strated goals. Euw jn connotes using knowledge and skills to liberate individuals 
from the bonds of ignorance— to expand her or his options; it is a process done primarily 
in the interest of the person being educated. Training seems to connote using the learn- 
ing process to moid and control individual behavior toward specif»c goals/roles that are 
needed by society; it results in conforming behavior, (p. 20) 

Nadler (1979) notes a similar distinction between training and education He defines training 
as "those activities which are designed to improve human performance on the job the employee is 
presently doing or is being hired to do" (p. 40). On the other hand, he defines education asjlhose 
"human resource development activities which are designed to improve the overall competence of 
the employee in a specified direction and beyond the job held" (p. 60). In essence, training per- 
tains to specific, narrowly defined, job-related learning experiences; whereas education pertains to 
learning experiences that develop the individual in a broader sphere, enabling him or her to take 
responsibility rather than simply fitting in as a cog. 

Thus, from a philosophical standpoint, short-term programs can be conceptualized as 
emphasizing training. This does not imply that their only objectives are aimed at developing nar- 
rowly defined, job-specific competencies. The United Auto Workers/Ford Targeted Vocational 
Retraining programs are an example of individual-oriented training in which skill training is a part 
of a more comprehensive program directed at developing new career ladders for laid-off workers. 
On the other hand, longer programs do not necessarily imply education of the individual in a 
broader sphere. Many certificate programs are more specific and are directed toward developing 
job-specific skills. Thus, although a certain amount of overlap may exist between short- and long- 
term training, they are, from a philosophical perspective, fundamentally different. What long- and 
short-term training have in common is that they strive to maximize their impact on productivity in 
the labor market while maintaining a high degree of efficiency in producing their respective prod- 
ucts. Just what that impact is— whether it is measurable or comparable between programs and 
over occupations and industries, and whether alternative policies and programs based on different 
philosophies would be more effective in producing desirable economic goals— is a question for 
evaluation research. 



Organizational Context 

Short-term training programs are offered in a variety of organizational contexts, each of which 
has different emphasis. Providers of short-term training include, for example, vocational-technical 
institutes, community colleges, employers, labor unions, joint labor union-management groups, 
the military, trade associations, various Federal and State governme r# agencies such as employ- 
ment agencies, welfare agencies, vocational rehabilitation agencies, and community agencies 
Their differences are likely to be expressed in their philosophies and purposes, their methods of 
selecting trainees, their training design, and tneir evaluation needs. 

Many short-term training programs are contracted or subcontracted out to training deliverers 
Each new organization involved in providing training brings with it its own policies and proce- 
dures. This additional structure is intended to assure overall program quality in the institution 
Short-term training contracts often modify or even short-circuit the impact of such institutional 
policies. 
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Long-term training programs, in contrast, are generally offered by 2- and 4-year colleges and 
universities, vocational-technical institutes, or labor unions and trade organizations Thus, the 
organizational context of the longer-term training programs is generally less complex than that of 
short-term skill training. 



Purpose 

Both short- and long-term training programs promote investment in human resources and 
economic development. Both types of program support the expansion and diversification of the 
local, State, and Federal economic bases by improving their employment and tax bases. Both pro- 
grams also support the retention and revitalization of existing businesses and industries and pro- 
vide skilled workers for new firms. 

Short-term training programs, however, are specifically directed toward meeting the imme- 
diate and unique labor force needs of a particular community. Paulsen (1981) notes that these 
programs "may be designed to meet immediate needs of business and industry for skilled workers, 
attract new companies to an area, or respond to State/Federal/professional licensirg or certifica- 
tion requirements" (p. 3). Long-term programs, on the other hand, are designed to meet future 
labor force needs. 

Long-term programs may also be aimed at meeting current structural imbalances in the labor 
market, especially in skill areas that require longer training. In instances of critical shortage of 
skills that take a longer time to develop, short-term training programs often provide temporary 
solutions until fully qualified workers become available. (The ultimate purpose of these short-term 
programs may be to "self-destruct" or evolve into longer programs, although economic conditions 
or various bottlenecks may stall this process.) 



Clientele 

By and large, the clientele of short-term training programs is different from that of long-term 
programs. Short-term program participants are usually sponsored by their employers or by agen- 
cies. Employer-sponsored particip 1 * As are commonly paid while they learn. In contrast, partici- 
pants : n long-term programs are usually self-selected volunteers who are personally committed to 
improving their future through training and education. Typically, they are not sponsored 

Many short-term training programs are designed for the disadvantaged and handicapped Skill 
training for the economically disadvantaged and retraining for dislocated workers are common, 
especially in training programs sponsored by government agencies (e.g., JTPA programs) Most 
public funding for economically disadvantaged populations is not sufficient for the long-term 
development of these groups. 



Instructional Staff 

Paulsen (1981) finds that, because of the short-term, sporadic nature of these skill training 
programs, postsecondary institutions often have difficulty with faculty availability, turnover, and 
instructional quality. He states that constant recruiting efforts result in higher costs for the 
institution 
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In field site visits, National Center researchers found that colleges and technical institutes re- 
cruited instructors for short-term programs from the ranks of their existing full-time and part-time 
staff, as well as directly from business and industry. In colleges accustomed to providing short- 
term training, program coordinators have built a sizable pool of instructors who can teach specific 
programs when they are offered. 

Some short-term programs are developed using existing courses, finding instructional re- 
sources for these programs is not difficult In other cases, however, as when a company needs 
state-of-the-art training in a specialized skill area, neither existing courses nor available instructors 
can provide appropriate instruction in the content area. In these cases, technical experts from 
within the company are sometimes used to provide instruction. These instructors may be given 
technical support in designing curriculum and instructional techniques by the postsecondary 
institution 

Instructors at Monroe Community College in Rochester, New York, have on occasion been 
paired with company technical experts during an initial sequence of teaching. This has allowed the 
college's instructors to learn the subject matter while the technical experts gained instructional 
skills. In this way, both the college and the companies involved have gained expertise that may be 
useful in designing and conducting future training. 

Ensuring the quality of instructors is important, particularly when a program is customized for 
individual businesses. Wenig and Wolansky (1983) surveyed employers' perceptions of outstand- 
ing employer-sponsored skill training and found that well-qualified trainers were among the cri- 
t' na seen as essential for effective training. Qualifications, however, are only one of the elements 
in a trainer's effectiveness; a multidimensional approach to trainer evaluation seems most appro- 
priate. This view is supported by the findings of a comprehensive study on trainer effectiveness by 
Bennett and L.educhowicz (1985) sponsored by the Manpower Service Commission of the United 
Kingdom. The researchers conclude: 

A unified concept of trainers does not exist and there do not appear to be a limited 
number of perspectives of trainers' effectiveness. Trainers' effectiveness is viewed in 
terms of a large number of factors which can be grouped under six major aspects— 
trainer role orientation and perception, trainer competencies, trainer characteristics and 
credibility, trainer work behavior and style, outcomes of trainer activity, and organiza- 
tional factors, (p. 42) 

Selection of instructors for short-term programs at postsecondary institutions is based not 
only upon their credentials as instructors and technical expertise, but also upon their experience 
and reputation within the college or institute. Administrators interviewed for this study said that 
instructors should be able to adapt to the culture of the business world and to be flexible, adapting 
and changing programs as needed. In fact, short-term programs may require very good instruc- 
tors. "Give me your A Team** was the mandate given to college short-term training coordinators by 
the executive director for Finger Lakes (New York) Regional Education Center for Economic 
Development. Some program administrators frequently monitor or evaluate short-term program 
instructors, particularly when the instructors are being used for the first time. 

Instructional staff for long-term programs, on the other hand, tend to be full-time employees 
of their organizations. In postsecondary institutions, these instructors may even be tenured Their 
primary responsibility is to deliver instruction. Since these instructors are full-time, their turnover 
rate is lower than that of their counterparts in short-term programs 
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Institutionalized, long-term skill training programs are usually designed on the basis of occu- 
pational analysis data, whereas short-term skill training in business and industry is generally 
designed from job and task analysis information. Occupational analysis data suggest generic skills 
common to various jobs within an occupation. Job analysis and task analysis result in descriptions 
of job competencies that are highly specific and refer to tasks performed in a particular enterprise 
at a particular point. 

Short-term training programs m ust often deal with the unique needs of the clientele, who may 
have very different entry-level competencies, cognitive styles, and learning styles For this reason, 
short-term programs are designed with great precision in order to lead to the desired performance 
outcomes. Because programs are often employer-sponsored, cost-effectiveness is frequently 
emphasized. Although outcomes and cost-effectiveness are also important in long-term training 
programs, these factors seem more critical to short-term training programs. Short-term training 
curricula are often systematically designed to reduce the probability of error in the curricula, thus 
enhancing program outc ;mes and cost-effectiveness. 

Short-term program curricula are more flexible in ature, content, duration, and format than 
long-term programs. However, this flexibility is costly. Many of these programs are "one-shof 
efforts, often tailor-made to meet one client's specific needs and not repeated. Therefore, recovery 
of start-up costs is usually not possible. Moreover, short-term programs offered by postsecondary 
institutions are often noncredtt, so tuition is not reimbursed by the State. 

The advantages to clients of purchasing training services from colleges or technical institutes 
often outweigh the additional cost of the training. Clients have the institution's assurance of qual- 
ity and can handpick courses from existing programs to meet their specific needs. Because facili- 
ties, and often specialized equipment, are already set up, the company does not have to develop or 
maintain them. Colleges can also gain by offering short-term twining Instructors can update their 
knowledge of skill training content while the institution develops good community relations 



Short-term skill training differs from long-term training in its commitment toward employment 
and in employer participation and involvement. Since short-term training is directly related to 
getting, maintaining, or upgrading employment, it must be designed to address particular needs 
Long-te r m programs often have only a general relationship to specific employer or trainee 
employment needs. Often, the only interface with the world of work is the placement office, col- 
legewide employer surveys, or vocational program advisory councils. These sources, though use- 
ful, generally do not directly affect the relationship of trainee and trainer and often they have little 
effect on curriculum, instructional evaluation, and other program components. Short-term pro- 
grams, on the other hand, exist because of their strong interface with the world of work. By neces- 
sity, they have a much higher degree of collaboration and linkage with employers at all levels of 
p r ogram administration than long-term programs have. 

A summary of the characteristics of short-term and long-term training programs hy cate- 
gor y\ distinctions is presented in table 1 . As stated earlier, there are some overlaps among char- 
acteristics and between the two types of programs. Identifying fundamental differences between 
short- and long-term training programs, however, may suggest elements to consider in designing a 
short-term program. 



Linkage with Employers 
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TABLE 1 

SUMMARY r r SHORT- AND LONG-TERM 
TRAlNiNG jGRAM CHARACTERISTICS 



Categorical 
Disttnctions 


Training Program 


Short-Term 


Long-Term 


Duration 


• A few hours to less than 1 year 
(generally not more than 6 months) 

• No diploma awarded 

• Some recognition "certificate" 


• More than 1 year 

• Diploma or 1-year certificate 


Philosophy 


• Emphasis on "training" 

• Goal oriented toward achieving 
specific objectives 

• Job-related learning experience 
specific objectives 

• Individual-related learning experience 
targeted toward specific objectives 


• Emphasis on "training" or "education" 

• Promotes excellent opportunities for 
growth and development 

• Objectives are less specific, more 
flexible 

• Knowledge for liberating individuals 

• Individual-related learning experiences 


Organiza- 
tions' 


• Providers of training may include' 

— vocational-technical institutes, 

— community colleges, 

— employers, 

— labor unions, 

— community organizations, 

— employment agencies, 

— vocational education secondary, 

— welfare agencies, 

— vocational rehabilitation, 

— JTPA, and 

— trade organizations. 

• Multiple forms of interagency arrangements 


• Providers generally include: 

— 2- ano 4-year colleges/universities 

— vocational technical institutes 

— labor unions 

• Training institution independent 


Purpose 


• Promotes investment in human resources 

• Meets immediate and unique labor force 
needs 


• Promotes investment in human resources 

• Meets future perceived labor force need? 

• Reduces structural unemployment 


Clientele 


• Participants often sponsored by their 
employer or by governmental agency 

• Generally experience little or no loss 
of wages as a result of participating 
in short-term training 


9 Smaller percentages of participants are 
sponsored by an employer or agency 
Generally pay for their own training/ 
education 

• May forgo some earnings in order to 
participate in long-term training 


Instructional 
staff 


• Are usually experts hired from the fie'd 

• Jobs characterized by high staff turnover 

• Does not consider teaching primary 
responsibility 

• May be full- or part-time instructors 

• Are selected from a pool of quahfieo 
instructors 


• Are usually full-time instructors at 
the institution 

• Is characterized by low turnover 

• Has teaching as primary responsibility 
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TABLE 1 - Continued 



Categorical 
Distinctions 


Training Program 


Short-Term 


Long-Term 


Curriculum 


• Is characterized by systematic 
instructional design 

• Includes a need assessment focused on 
specific problems 

• Focuses on client needs 

• Has a specific narrow emphasis on depth 

• Focuses on the immediacy of application 
of acquired skill 

• Has specific behavior objectives 

• Offers little freedom and flexibility for 
trainer in terms of objectives/curriculum 

• Usually produces a comprehensive student 
guide, workbook, and instructor's manual 

• Intent is specific 

• Accountability is emphasized 

• Emphasizes mastery of skills 

• Competency-based training 

• Usually offered as noncredit 

• May be linked with reguiar programs 


• is based on occupation analysis 

• Usually reduces needs assessment to 
employment outlook research 

• Focuses on individual and societal 
needs 

• Emphasizes breadth 

• Focuses on future perceived needs 

• Has general objectives 

• Provides instructor with greater freedom 
and flexibility in terms of curriculum 
objectives 

• Has textbook and laboratory manuals 

• Usually has interdisciplinary content 

• Does not overemphasize accountability 

• Supports mastery of principles and 
skills 

• Usually is not competency-based 

• Is offered for earned credit 


Linkages 
with 

Employment 


• Programs generally hrked with 
specific employment 

• Greater percentage of resources on 
employment linkages 


• Linkage with employment more general 

• Relatively fewer resources devoted to 
employment linkage 
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CHAPTER 3 



PROGRAM EVALUATION: AN OVERVIEW 



This chapter reviews major program evaluation models end approaches that have poter.iial for 
the evaluation of short term training programs First, however, it is necessary to define what is 
meant by program evaluation. 

The literature offers various definitions of program evaluation, and many would agree with 
Ruthman (1984) when he argues that "program evaluation has no uniform and consistently applied 
definition. The term has become the subject of a variety of interpretations in relation to its pur- 
poses, scope and methodology" (p. 10). 

Tyler (1942) was among the first to define evaluation, which he did in connection with the 
Eight-Year Study at The Ohio State University. Evaluation, he decided was the process of deter- 
mining whether the objectives of a program had been achieved. The Phi Delta Kappa National 
Study Committee on Evaluation (Stufflebeam et al. 1971) defined education evaluation in more 
detail as "the process of delineating, obtaining, and providing useful information for judging deci- 
sion alternatives" (p. 40). The Phi Delta Kappa Committee identifies eight key terms in this defini- 
tion (as italicized above). They are as follows: 

• Process— A particular and continuing activity subsuming many methods and involving a 
number of steps or operations 



• Delineating— Identifying evaluative information required through an inventory of the deci- 
sion alternatives to be weighed and the criteria to be applied in weighing them 

• Obtaining— Making information available through such processes as collecting, organiz- 
ing, and analyzing and through such means as measurement, data processing, and statis- 
tical analysis 

• Providing— Fitting information together into systems or subsystems that best serve the 
purposes of the evaluation and reporting the information to the decision maker 

• Useful— Satisfying the scientific, practical, and prudential criteria and pertaining to the 
judgemental criteria to be employed in choosing among the decision alternatives 

• Information— Descriptive or interpretive data about entities (tangible) and their relation- 
ships, in terms of some purpose 

• Judging— The act of choosing among the several decision alternatives; the act of decision 
making 

• Decision alternatives— Two or more different actions that might be taken in response to 
some situation requiring altered action (pp 40-43) 
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Another definition o* evaluation is proposed by Worthen and Sanders (197C) 



Evaluation is the determination of the worth of a thing. It includes obtaining information 
for use in judging the worth of a program, product, procedure, or objective, or the poten- 
tial utility of alternative approaches designed to attain specified objectives, (p 19) 

In a similar vein, the Joint Committee on Standards for Educational Evaluation (1981) defines 
evaluation as the systematic assessment of the worth or merit of some object. 

Berk (1981) argues that the literature offers many different definitions of evaluation (eg , Alkin 
1972, Cooley and Lohnes 1976; Cronbach 1963; Fink and Kosecoff 1980; Freeman 1977; Guba 
1966, Popham 1975; Posavac and Carey 1980; Provus 1967, 1971; Scriven 1967, Stake 19S7, 
Stufflebeam et al. 1971; Walberg 1974; Wolf 1979). For his own purposes, Ruthman (1984) defines 
program evaluation as "the use of scientific methods to measure the implementation and out- 
comes of programs for decision-making purposes." A careful analysis of these definitions shows 
that they agree on one point— that the purpose of evaluation is to provide information for decision 
making. 



Evaluation Models 

Most evaluation theorists and practitioners use models to conceptualize program evaluation, 
and there has been a proliferation of models in recent years. However, a careful analysis of 
numerous recent models shows that they differ only in minor respects from the established 
models, which have acquired widespread acceptance through extensive field testing. The newer 
models tend to be adaptations of established models that are tailored to meet specific needs. 

Stake (1967), Popham (1975), and Worthen and Sanders (1973) present and clarify many of 
these major models. Drawing upon their work, House (1978) has developed a taxonomy of the 
major approaches to program evaluation. His classification provides a useful framework for identi- 
fying those models, or approaches, that might be useful in assessing sho»t-term skill training pro- 
grams. The models he describes are systems analysis, behavioral objectives, decision making, 
goal-free evaluation, art criticism, accreditation, and transaction. The following sections briefly 
describe each of these models and the major questions they address. 

Systems Analysis Model 

The systems analysis model is rooted in the principles of scientific management. In this 
approach, education is perceived as "a process of social service production. ... its aim is to 
rationalize the educational process in order to maximize efficiency [i.e., increase productivity and 
reduce waste]" (Farley et al. 1985). Farley et al. further identify the following underlying assump- 
tions of the systems analysis approach: 

• Program goals and objectives can be explicitly stated in measurable (i.e., quantifiable) 
terms. 

• The use of scientific methods through controlled experiments can uncover constant and 
uniform relationships between system "inputs" and system "outputs " 
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• The findings from such studies can be used to develop formal procedures by means of 
which schools and teachers can efficiently produce the desired level of program objec- 
tives, (p 26) 



The systems analysis mode, is useful in answering questions about efficiency. "Are the 
expected effects achieved? Can the effects be achieved more economically'? What are the most 
efficient programs?" (House 1978, p. 12). The model actually uses correlational or experimental 
techniques to analyze the effects of alternative production functions 



Behavioral Objective Model (BOM) 

The behavioral objective evaluation model was one of the first to be introduced and is still the 
model most commonly used by practitioners. Introduced by Tyler in the 1940s and 1950s, this 
model brought a scientific management orientation to evaluation. Farley et al (1985) note that 
Tyler advocated a behavioral objective approach to curriculum design and that he further empha- 
sized the need for defining objectives "in terms of ihe behavioral outcomes students would be 
expected to demonstrate upon completion of the curriculum" (p. 56). 

House (1980) provides the following brief summary of the behavioral objectives model of 
evaluation. 

The evaluation of a program should define its outcome and its objectives in specific 
individual behaviors. The task of the evaluator was to determine whether the students 
were exhibiting these particular behaviors after being in the program, (p. 26) 

Objectivity and quantitative measurement are two key characteristics of this kind of evalua- 
tion. Although initially designed for curriculum design and revision, this model, the behavioral 
objective approach, has also been widely used for evaluating educational and social programs 

According to Tyler's model of program evaluation, the behavioral objectives approach 
includes the following steps: (1) formulate goals and objectives, (2) define these objectives in 
behavioral terms, (3) specify ideal situations for the achievement of these objectives, (4) select 
appropriate measures, (5) measure the achievement of the objectives, and (6) compare the 
achievement with the previously established criteria. In this approach, goals that are not attained 
reveal inadequacies in the program; the program is considered successful when the goals are met. 

Hammond (1973) designed a model of evaluation that includes five basic steps* (1) isolate the 
subject to be evaluated in the program, (2) define the institutional and instructional variables, 
(3) formulate objectives in behavioral terms, (4) assess the behavior identified in the objectives, 
and (5) analyze the results of the assessment. 

Metfessel and Michael (1973) propose yet another model wi*h strong emphasis on behavioral 
objectives. Eight main steps are identified in their approach to evaluation 

1. Involve members of the total community 

2. Construct broad goals and specific objectives 

3 Translate specific objectives into forms that are communicable and that facilitate learning 
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4 Develop instrumentation 



5 Carry out periodic measurement 

6 Analyze data collected 
7. Interpret data analyzed 

8 Formulate recommendations for the modification of programs, goals, and objectives 

The most useful contribution of this model is that it provides for criterion measures against which 
the actual achievement can be compared to determine if the objectives have been achieved 

Decision-Making Model (DMM) 

The decision-mr' ing evaluation model reflects the belief that evaluation should generate 
information that can facilitate intelligent judgement by decision makers. The context, input, pro- 
cess, and product model (CIPP) designed by Stuff lebeam et al. (1971) has received widespread 
acceptance and has been extensively used. 

Stufflebeam and Shinkfield (1985) provide an updated version of their definition of evaluation 

Evaluation is the p r ocess of delineating, obtaining, and providing descriptive and 
judgemental information about the worth and merit of some object's goals, design, 
implementation, and impacts in order to guide decision making, serve needs for account- 
ability and promote understanding of the involved phenomena, (p. 159) 

Stufflebeam and Shinkfield (1985) also indicate the objectives of the four types of evaluation 
included .n the CIPP model: 

• Context evaluation— To define institutional context, to identify the target population and 
assess their needs, to identify opportunities for addressing the needs, to diagnose prob- 
lems underlying needs, and to judge whether proposed objectives are sufficiently respon- 
sive to the assessed needs 

• Input evaluation— To identify and assess system capabilities, alternative program strate- 
gies and procedural designs for implementing the strategies, budgets, and schedules 

• P/ocess evaluation—To identify or predict defects in the procedural design or its imple- 
nentation while implementation is ir Process, to provide information for the prepro- 
grammed decisions, and to record or judge procedural events ind activities 

• Product evaluation— To collect descriptions and judgments of outcomes and to relate 
them to objectives and to context, input and process information, and to interpret their 
worth and merit (p. 170) 

In the most recent version of the CIPP model, Stufflebeam and Shinkfield (1985) suggest 
alternative methods of data collection for each type o\ evaluation: 

• Context evaluation— Using such methods as systems analysis, survey, document review, 
hearings, interviews, diagnostic tests, and the Delphi technique 
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• Input evaluation— Inventorying and analyzing available human and material resources, 
solution strategies, and procedural designs for relevance, feasibility, and economy, litera- 
ture search; visits to exemplary programs; advocacy teams, and pilot trials 

• Process evalrMion— Monitoring the activity's potential procedural barriers and remaining 
alert to unanticipated barriers; obtaining specmed information for programmed decisions, 
describing the actual barriers, and interacting with and observing the activities of project 
staff 

• Product evaluation— Defining operationally and measuring outcome criteria, collecting 
judgments of outcomes from stakeholders, and performing both qualitative and quantita- 
tive analyses (p. 171) 

Stufflebeam and Shinkfield (1985) also introduce two distinct roles for the CIPP evaluation 
models, namely decision making (formative orientation) and accountability (summative orienta- 
tion) However, they indicate that the major emphasis of the model is still on the decision-making 
process. 



Goal-Free Evaluation Model 

Scriven (1983) is a respected evaluation theorist whose theories have brought important 
ins«ghts into the process. Scriven was the first evaluation theorist to make the distinction between 
goal-based and goal-free evaluation. In goal-based evaluation the evaluator examines only the 
program goals and assesses the extent to which these goals are achieved. In goal-free evaluation, 
the evaluator focuses on the unanticipated as well as on the intended outcomes of a program 
Explicating his advocacy of goal-free evaluation, Scriven notes: 

Goals are often best seen as inspirational devices— they make poor foundations for 
analysis. It is also important to note that for evaluators to be aware of the goals of a pro- 
gram is for them to be given a strong perceptual bias in a particular direction which, in 
conjunction with whatever positive or negative effect they possess for a program, 
unleashes the possibility of distorted perception of the rasults (p. 237) 

Popham (1975) points out that "the chief advantage of goal-free evaluation is that it encour- 
ages the evaluator to be attentive to a wider range of program outcomes than might be the case 
with a goal-based evaluator who has been influenced to look at project results consonant with proj- 
ect aims" (p 28). 

Although Scriven (1983) advocates a goal-free approach to evaluation, he sees goal-based and 
goal-free evaluation as complementary. An evaluation can start as goal free, and then shift to goal 
based, or both approaches can be used simultaneously by different evaluators. 



Art Criticism Model 

The art criticism model as used in evaluations of educational programs involves the use of 
expert judgements of the worth or merit of an educational progrr tv House (1978) describes this 
model as "the model of an educational critic, one who has attained, by experience and training, the 
ability to judge the important facets of educational programs" (p. 5). The art criticism model 
assumes that the major objective of evaluation is to improve the overall standards of programs, 
and it further assumes agreement upon standards and the qualifications of critics. 
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Accreditation Model 



The accreditation model is the primary tool used by the North Central Association for accredi- 
tation of programs. The model has been adopted by many States for use in their program review 
process The review process involves an internal evaluation of the program by local administrators 
and instructional staff according to predetermined criteria, standards, and procedures An external 
panel of experts then conducts an on-site evaluation using the same standards and criteria The 
Minnesota Adult Vocational Education Evaluation Systam, Ohio's Program Review for Improve- 
ment, Development, and Expansion of Vocational Education (PRIDE), and the Arkansas Vocational 
Evaluation System are examples of State vocational evaluation programs that use this model 



Transaction Model 

This model uses primarily qualitative approaches and focuses on the educational process 
Case studies, interviews, and observations are used to expiore the activities that take place in the 
educational programs as they appear to different actors. 

Stake (1967) defines these key components in his transaction model. 

• An antecedent— A condition existing prior to teaching and learning that may relate to 
learning outcomes. The status ot the student prior to the lesson (i.e., the aptitude, pre- 
vious experience, interest, and willingness) is a complex antecedent. 

• Transactions— Tie encounters of students with teachers, student with student, author 
with reader, parent wivh counselor— the succession of engagements that comprise the 
process of education. Examples are the presentation of a film, a class discussion, the 
working of a homework problem, an explanation on the margin of a term paper, and the 
administration of a test. 

• Outcomes— A body of information that includes measurement of the impact of instruction 
of teachers, administrators, counselors, and others. The wear and tear on equipment, 
effect of the learning environment, and required costs are also outcomes. Outcomes con- 
sidered in Stake's transaction model of evaluation include not only those that are evident, 
but also applications, transfers, and relearning effects that may not be available for mea- 
surement until long after a learning session has ended, in short, outcomes are the conse- 
quences of educating— immediate and long range, cognitive and conative, and personal 
and public. 

The Stake model of evaluation calls for the identification of the rationale of the program being 
evaluated. Stake (1967) says, "The rationale should provide the basis for evaluating intents" (p 9) 
Stake feels that "there are two bases for judging the characteristics of a program: (1 ) with respect 
to absolute standards as reflected by personal judgments and (2) with respect to relative standards 
as reflected by the characteristics of alternate programs" (p. 13). 

All the evaluation models just described differ in their target audiences. They are based upon 
different methodologies and different assumptions of what is true, known, and knowable. They are 
similar, however, in that each presents a way of systematically assessing the merits or worth of 
training programs. 
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Evaluation Design 



An evaluation design is a systematic plan for learning the answers to a specific set of ques- 
tions about a training program. The term design is often used loosely in the 'iterature of program 
evaluation to mean two distinct things, first, a plan for conducting evaluation, and second, the 
research design after which an evaluation activity is patterned For the sake of c'anty, this guide- 
book refers to the latter as the evaluation research design 



Types of Evaluation Design 

Brinkerhoff et al. (1983) identified six main types of evaluation designs 

• Fixed evaluation 

• Emergent evaluation 

• Formative evaluation 

• Summative evaluation 

• Experimental and quasi-experimental evaluation 

• Unobtrusive inquiry 

An evaluation design is fixed when the process is systematically planned in advance and all 
the procedures for data collection, analysis, and reporting are spelled out in detail. The emergent 
design, on the other hand, "readily responds to ongoing influences, evolving as it accommodates 
cnanging audiences, problems and program activities. Emergent evaluation designs are usually 
>sed in conjunction with naturalist inquiry" (Lincoln and Guba 1985, p. 4). 

Because of the different roles imposed on formative and summative evaluation, distinct evalu- 
ation designs are often needed to satisfy clients' needs for information. In commenting on this 
issue, Brinkerhoff et al. (1983) note: 

It is possible to build evaluations to provide ongoing information for improvement and 
information for judgments of worth. However, often formative and summative evaluation 
have conflicting purposes and cannot easily be reconciled in the same design, (pp. 
38-39) 

In the past, program evaluation has been dominated by the paradigm of scientific inquiry that 
draws its roots from the natural and agricultural sciences. The emphasis has been on close manipu- 
lation and close monitoring of treatment under control conditions. Program evaluation designs of 
this kind were patterned after experimental and quasi-experimental research (see Campbeil and 
Stanley 1963), and their ultimate goal was the establishment of a causal relationship between the 
training program and its outcomes. In scientific inquiry, the strength of an evaluation design is 
judged by the rigor of the manipulation and control procedures. However, the scientific paradigm 
has failed to provide decision makers with answers to complex questions involving complex, 
situation-specific human interactions and values. Scholars in education have therefore investi- 
gated alternative modes of inquiry. This new vision of program evaluation has resulted in *he 
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naturalistic paradigm All the elements that are viewed as weaknesses in scientific inquiry consti- 
tute the strengths of naturalistic inquiry (e g , the absence of control and manipulation, the mclu 
sion of contextual factors, tl.s attention to values, and the interaction of the evaluator as a human 
instrument). 

The major focus of scientific inquiry is to find significant statistical differences between experi- 
mental and control groups. Naturalistic inquiry, on the other hand, attempts to provide a tmck 
description of reconstructed, multiple realities in a particular setting through in-depth case study 
methods Naturalistic inquiry makes use of an emergent design that unfolds with the progression 
of the inquiry process (Lincoln and Guba 1985). 

Two other major differences that characterize the design of scientific inquiry are (1) random 
selection of subjects and (2) randomized treatment of groups. In the true experiment in the sci- 
entific mode, many threats to internal and external validity are controlled by random selection of 
subjects, random assignment of subjects to groups, and random assignment of treatments to 
experimental and control groups. This is often constraining, as random selection is not always 
feasible, nor desirable for ethical reasons; it might, for example, entail denying training opportuni- 
ties to some individuals Another major drawback of true experimental design is the artificial situa- 
tion it creates. The quasi-experimental design that makes use of intact groups is less constraining 
and is often preferred by educational evaluators. Naturalistic inquiry, on the other hand, makes use 
of intact groups in natural settings and does not control and manipulate treatments. This limits the 
generalizations that can be made, but the thick description resulting from a naturalistic inquiry 
does make provision for transferring conclusions to similar settings. 



The evaluation standards established by the Joint Committee on Standards for Educational 
Evaluation (1981) may be adopted to serve as criteria for the evaluation of short-term training pro- 
grams. The Joint Committee identified four attributes of every program that should be evaluated, 
utility, feasibility, propriety, and accuracy. The Joint Committee stated, "The Committee is satis- 
fied that standards which shape an evaluation so that it has these four characteristics are neces- 
sary and sufficient for sound evaluation in education" (p. 13). 

The Joint Committee identified 30 standards that together constitute the 4 characteristics Fif- 
teen of the 30 criteria appear to have the most relevance for short-term skill training programs, and 
they are described in the following sections. 1 



Information scope and selection. The information collected for an evaluation should be suffi- 
cient to support a judgment of the program's worth and merit. The information is useful when it 
answers relevant program evaluation questions. 

Clarity of information. The information collected should relate clearly and specifically to the 
program evaluation objectives. The information should provide a firm foundation for evaluation 
conclusions and recommendations. Finally, the information should be characterized by logical 
development so that the program evaluation audiences can understand it. 



'This information is taken from McKinney and Kohan (1905. pp 21-33) 



Evaluation Criteria 



Evaluation Standards 



Utility 
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Timeliness. The evaluation should provide information to the program developers and spon- 
sors at the time when the information can best be used The most critical information needs of the 
program developer and sponsor must be met on time to avoid delays in important decisions about 
the program 



Practical procedures. Program evaluation procedures should be realistic and should reflect 
time and financial constraints. Procedures should take into account practical problems If, for 
example, the evaluation is formative in nature, procedures for collecting information should be 
designed to minimize school and classroom disruptions. 

Political viability. The evaluation procedures should be politically viable given the various 
interest groups and stakeholders that surround the program. Procedures are politically viable 
when the purpose of the evaluation is achieved despite pressures from the formal and informal 
organizational power structures of the community, sponsor, and client. 

Cost-effectiveness. The evaluation should produce information of sufficient value to justify the 
expense of conducting the evaluation. Alternative evaluation approaches that might produce more 
useful information at the same or less cost should be examined. In summary for the information 
needed, the evaluation should be conducted as economically as possible. 



Rights of human subjects. The evaluation should respect and protect the rights and welfare of 
human subjects Legal, ethical, and common sense issues should be carefully examined. 

Conflicts of interest. The program evaluation should be designed to avoid conflicts of interest, 
which can compromise the evaluation process and results. 

Balanced reporting. The evaluation should be complete and fair in its presentation of the pro- 
gram's strengths and weaknesses, reporting both negative and positive findings Strengths and 
weaknesses should not be manipulated to please partisans or interest groups. Findings that might 
prove embarrassing to some groups or individuals should not be omitted 



Clearly identified object. The object of the program evaluations should be clearly identified 
and realistically described. Unique features of the object should be identified. Furthermore, the 
descriptions and unique features of the object should be a valid characterization of the object 

Appropriate information sources. The program evaluation should provide adequate informa- 
tion to answer the evaluation questions. If possible, multiple information sources should be 
tapped, using a variety of methods, such as interviews, surveys, and observations. 

Adequacy of contextual Information. Contextual information should be used in interpreting 
program evaluation results. For example, the program developer and client should know whether 
the program's success or failure was influenced by the academic background or socioeconomic 



Feasibility 



Propriety 



Accuracy 
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status o< the participants, the instructor's background, the classroom climate, instructor a;id spon- 
sor support or resistance of the program, and/or community support or apathv toward the 
program. 

Program and procedures explicated. The purposes and procedures of an evaluation should be 
understood by those involved. The objective of the evaluation should be clear The procedures for 
collecting, organizing, analyzing, and reporting should be described in such detail that other evalu- 
ators could replicate the evaluation effort. 

Reliability. In data collection activities, reliability refers to accuracy in measurement Although 
there are several types of reliability (e.g., stability, interrater, equivalency, internal consistency), all 
refer to determining error. The less error in a measurement, the more reliable it is. When doing an 
evaluation, the reliability of the measuring instruments must be discussed, and reliability in general 
must be addressed. 

Validity. Validity refers to "how truthful, genuine and authentic data are in .epresentmg what 
they purport to. To be valid is to make truthful claims, instruments must measure what they intend 
and claim to measure" (Brinkerhoff et al. 1983, p. 100;. As with reliability, there are different types 
of validity (content, concurrent, predictive, and construct). No matter which type of validity is 
applicable to the evaluation effort, the validity of the data collection instruments must be 
addressed. 



Attesting Evaluation Approachet 

No single evaluation approach can be expected to satisfy all 15 criteria, and it is important to 
know what criteria a given approach does or does not satisfy. One way you can examine evalua- 
tion approaches in view of these citeria is shown in figure 1, a sample evaluation instrument The 
left-hand column of the table lists the criteria just discussed, and each of the next four columns is 
headed by one of the most widely used approaches to evaluating short-term training. You can rate 
how well a given approach meets each criterion by circling the appropriate number where the line 
and column intersect. The rating key explains the meaning of the numbers. 

Additional evaluation models or approaches may be included in this framework. The impor- 
tance of the framework is to underscore the idea that short-term training program evaluation 
design and assessment should meet a comprehensive criterion set. The relative importance of 
each criterion may be determined by the evaluation sponsor, the client, or by a consensus of both 
these stakeholders. Additional criteria may be added to the framework by the key stakeholders in 
the evaluation effort. 
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Four Predominant Short-Term Program Evaluation Approaches 


Evaluation Criteria 


Needs Assessment 


Follow-up Studies 


Outcome Studies 


Cost-Benefit 
Cost-Effectiveness 


Utility 

— Information scope 

— Clarity of 
information 

— Timeliness 


1 2 3 

1 2 3 
1 2 3 


1 2 3 

1 2 3 
1 2 3 


1 2 3 

1 2 3 
1 2 3 


1 2 3 

1 2 3 
1 2 3 


Feasibility 

- Practical 
procedures 

- Political 
viability 

- Cost 
effectiveness 


1 2 3 
1 2 3 
1 2 3 


1 2 3 
1 2 3 
1 2 3 


1 2 3 
1 2 3 
1 2 3 


1 2 3 
1 2 3 
1 2 3 


Propriety 

— Rights of human 
subjects 

— Conflicts of 
interest 

— Balanced 
reporting 


1 2 3 
1 2 2 
1 2 3 


1 2 3 
1 2 3 
1 2 3 


1 2 3 
1 2 3 
1 2 3 


1 2 3 
1 2 3 
1 2 3 


Accuracy 

— Clearly identi- 
fied object 

— Appropriate in- 
formation sources 

— Adequacy of 
contextual 
information 

— Purposes and 
procedures 
explicated 

— Reliability 

— Validity 


1 2 3 
1 2 3 

1 2 3 

1 2 3 
1 2 3 
1 2 3 


1 2 3 
1 2 3 

1 2 3 

1 2 3 
1 2 3 
1 2 3 


1 2 3 
1 2 3 

1 2 3 

1 2 3 
1 2 3 
1 2 3 


1 2 3 
1 2 3 

1 2 3 

1 2 3 
1 2 3 
1 2 3 



Rating Key: 1 = Did not satisfactorily meet the criteria 

2 * Partially met the criteria 

3 = Satisfactorily met the criteria 



Figure 1. Sample evaluation instrument 




CHAPTER 4 



OPTIONS IN EVALUATION DESIGN 



Chapter 3 discussed some of the general questions that administrators need to consider in 
assessing short-term skill training programs. This chapter discusses the various components of 
evaluation and suggests guidelines for applying them to short-term skill training programs The 
components are treated as options from which the evaluator selects those that seem effective and 
appropriate to particular program evaluation needs. 



Evaluability Assessment 

The accountability movement has been the major force behind the widespread implementation 
o» program evaluation. It follows that if resources are to be committed to an evaluation, then the 
evaluation itself must show promise. The procedure used to ensure that an evaluation effort will 
tuitm its goats is referred to as evaluability assessment. The purpose of this assessment is to esti- 
mate the extent to which the goe's of an evaluation are likely to be met, "considering such factors 
as the program's characteristics, the evaluability research methodology, cost, and constraints on 
the use of desired research methods" (Ruthman 1984, p. 28). 

The fundamental questions addressed by an evaluability assessment are these. 

• Is there a demand for the evaluation? 

• What decisions are to be served by the evaluation activity? 

Most evaluation efforts go through four main phases or stages: needs assessment, program 
planning, formative evaluation, and summative evaluation. The need for an evaluation effort is 
indicated by either an internal source or an external source. As the information needs of internal 
and external agencies are not always the same, the types of evaluation required to address these 
varying needs may differ. As illustrated in figure 2, internal requests may indicate a need for all 
four types of evaluations, whereas the external requests, which generally come from funding 
agencies often focus on needs assessment and summative evaluations. An evaluability assess- 
ment should enable an administrator or an evaluator to determine whether it is appropriate to— 

• conduct a needs assessment, 

• describe how programs were planned and implemented, and how they operate (program 
planning), 

• provide information to program planners for program improvement (formative evaluation), 
or 

• provide accountability information to funding agencies (summative evaluation) 
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Needs 
Assessment 


Program 
Planning 


Foimative 
Evaluation 


Summative 
Evaluation 


iniernai nequcSl 
for Evaluation 


Yes 


Yes 


Yes 


Yes 


External Request 
for Evaluation 


Yes 
Sometimes 


Not 
Usually 


Not 
Usually 


Yes 
(Main Focus) 



Figure 2. Sources for initiation of program evaluation 



For many short-term skill training programs the initial contract plan or proposal spells out in 
considerable detail the required program evaluation activities, as well as specific criteria and 
standards for measuring and judging the program success. In these instances, service providers 
are bound to evaluate the program according to the initial evaluation plans. When such evaluation 
specifications are not formulated initially, the evaluator must negotiate with the agency requesting 
the evaluation in order to determine the purpose cf the evaluation effort. 

Four important factors should be considered when conducting an evaluabilKy assessment 
funding, human resources, equipment, and organizational support and feasibility 



Funding 

Evaluation is difficult to do well, and good evaluation can be expensive; therefore, the evalua- 
bility assessment must determine whether the sponsor of the evaluation J s committed to funding 
the evaluation. When evaluation is requested internally, a similar financial commitment is still 
necessary. At this stage, it may not br possible to estimate accurately the cost of conducting the 
evaluation, but an approximate estimate is nevertheless useful and will give a rough idea of the 
financial investment involved. The financial resources allocated to the evaluation activity influence 
the breadth, depth, and rigor of the evaluation and therefore the quality of the information gener- 
ated by the effort, as well as the credibility of the results. 



Human Resources 

Designing and conducting an evaluation calls for highly specialized skills and expertise. The 
success or failure of an evaluation effort, as well as the quality, usefulness, and credibility of the 
results, depends to a great extent on the quality ot the evaluator. Locating the appropriate individ- 
uals to design and conduct the evaluation activities should therefore be a major point of concern 
The basic qualifications of a good evaluator include the ability to— 

• communicate verbally and in writing; 

• solve problems; 
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practice interpersonal skills, 



• make detailed observations, 

• use computational skids, 

• write problem statement and evaluation proposals, 

• estimate evaluation costs; 

• conceptualize, design, manage, and conduct evaluation activities, 

• design questionnaires, achievement tests, and other data gathering instruments, 

• analyze qualitative data, interviews, and observation/field notes; 

• analyze evaluation results using computers and common statistical packages, 

• write precise evaluation reports, 

• give oral presentations of evaL iOn reports; 

• apply evaluation theories (both quantitative and qualitative), models, designs, and 

• conduct and design educational research; and 

• use descriptive and inferential statistics. 

When an internal evaluation has been requested, an individual who fits these qualifications 
should be identified within the organization. If no one with the appropriate skills and background 
is available, the alternative is to seek the help of private consultants. Many private firms specialize 
in evaluation, but their fees are often exorbitant. A more cost-effective alternative might be to hire 
an experienced university faculty member to supervise the effort. External agencies that commis- 
sion evaluations usually make use of their own evaluators or private consultants 

Equipment 

Depending upon the scope of the evaluation efforts, large amounts of data may be collected 
during the process. Efficiency in summarizing and analyzing these data and making the results 
available to decision makers is of primary importance if the results are to be useful in making 
timely decisions. Consequently, data processing equipment and statistical software are necessary 
for processing the information gathered. When achievement tests are administered to large 
numbers of students in an evaluation, considerable time can be saved by using an optical scanner 
to score answer sheets. In conducting the evaluability assessment, it is important to identify the 
data processing equipment that will be available to increase the efficiency and effectiveness of the 
evaluation process. 
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Organizational Support and Feasibility 



Successful program evaluation, whether internally or externally commissioned, requires the 
support of the entire organizational and administrative staff. It is best to determine at the onset if 
such support will be provided, and this can be ascertained during the evaiuaoihty assessment 

Another important consideration is the feasibility of the evaluation activity. This can be deter- 
mined through an analysis of the contextual constraints as well as the existing constraints on 
resources, materials, expertise, staff, and time. In an internal evaluation, the evaluabihty assess- 
ment can also help to determine the roles of key actors in the process (e.g , the evaluator, the 
developer, and the program planning team). 

The evaluabihty assessment is an important part of the evaluation and should take place 
before any major program evaluation effort. It can save scarce resources, time, and effort that 
otherwise might be lost in evaluation activities that fail to generate the information needed 



Needs assessment is the foundation upon which most training and development projects are 
built It is a "systematic management tool in both program planning and development" (McCaslm 
and Lave 1976, p. 5). It can be used to determine priorities and to facilitate the decision-making 
process (Witkin 1984). 

A need has been defined as "a gap between what is and what should be in terms of results" 
(Kaufman 1983, p. 3). Building upon his definition, Kaufman has formulated a comprehensive defi- 
nition of needs assessment in the context of his organizational element model. A needs assess- 
ment, he states, is ideally concerned with gaps in outcomes. It is a formal analysis that shows (and 
documents) the gaps between current results and desired results, arranges the gaps (needs) in 
order of priority, and selects the needs to be resolved. Needs assessment forms an integral part of 
program planning and evaluation, as illustrated in figure 2 shown earlier in this chapter. 

There is some debate as to whether needs assessment is part of program planning or program 
evaluation. In fact, it is both. Needs assessment is a prerequisite to program evaluation It is used 
to identify and assess these existing needs that must be addressed. After program implementation, 
needs assessment helps to establish whether and to what extent the needs have been met. This 
teads to decisions regarding program continuation, termination, or modification. 

Needs assessment and program evaluation are two processes that facilitate decision-making 
Although they use the same kind of information, their focus in the decision-making process is dif- 
ferent. Program evaluation operates at the program level, and needs assessment at the organiza- 
tional level (Witkin 1984). 

McCaslin and Lave (1976) advocate a four-step approach to needs assessment 

• Identify program goals or objectives that need to be assessed. 

• Identify procedure for determining and measuring the present status 

• Compare differences or discrepancies that exist between the goal and the actual status 

• Assign priorities or rankings to the discrepancies found, (p. 6) 



Needs Assessment 
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We believe that it is important to add the following step to the McCasIm and Lave approach 
because the needs assessment process is not complete without a needs analysis 



• Conduct a needs analysis to determine what factors in the system are responsible for the 
"what is" and "what should be" status of goals. 



Whom to Include 

When conducting a needs assessment it is important to determine whom to include in the 
process in order to get representative input from individuals or groups interested in, or affected by, 
the process. Needs assessment can focus on three levels of needs. (1) individual needs, (2) em- 
ployers' needs, and (3) societal needs. Individual needs may Include interests, knowledge, under- 
standing principles, skills, and attitudes. Employer needs may be considered in terms of such 
factors as industrial growth, occupational supply and demand, and present and projected em- 
ployment needs and conditions. 

Societal needs may be educational (providing an adequate education for all citizens), eco- 
nomic (promoting economic growth and development), social (providing the poor and the dis- 
advantaged with a means of moving beyond these barriers), and governmental. 

Witkin (1984) identifies three primary groups involved in needs assessment activities: (1) ser- 
vice providers (agencies, institutions, organizations, program developers), (2) service receivers 
(ihe beneficiaries of services), and (3) siakehoiders (agencies or organizations tnat nave an 
interest in the service received). Some groups that may be included in a needs assessment of 
short-term skill training are students, educators, employers, employees, government agencies, and 
professional organizations. 

Students. The needs assessment may include the following subcategories of students in the 
process. 

• Students currently registered 

• Graduates 

• Dropouts 

• The temporarily unemployed 

• The hard-core unemployed 

• The structurally unemployed 

• Dislocated workers 

• Welfare recipients 



Youth 



Adults 
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Educators. The following subgroups may be included in the needs assessment 



• Instructors 

• Administrators 

• Supportive staff 

• Regional education agency personnel 

• State department of education personnel 

• Faculty members from universities 

Employers and employees. A successful needs assessment of a training program in business 
or industry surveys both the employers and the employees. It may be meaningful to include indi- 
viduals at various levels of responsibility within the organization. 

Government agencies. This group is largely composed of stakeholders in the short-term skill 
training program. These might include. 

• State boards of education 

• Manpower councils 

• State departments of labor 

• State legislators 

• Departments of economics and community development 

• Industrial commissions 

• JTPA administrators 

• Private industry councils (PlCs) 

• State job training coordinating councils (SJTCCs) 

The purpose of the needs assessment effort, as well as practical considerations such as resource 
and time constraints, will influence the selection of individual groups or subgroups 



Procedures for Conducting Needs Assessment 

Various strategies can be used for gathering information for a needs assessment. Because of 
the values involved in the process, it is suggested that both qualitative and quantitative methods be 
used for collecting the data. Following are some of the most common methods in current usage 
surveys, interviews, nominal group techniques, the Delphi method, fault-free analysis, and 
decision-free techniques. Zemke and Kramlinger (1984) provide an excellent description of these 
methods and a discussion of the issues involved in their use, as well as an overview of highlights of 
the advantages and disadvantages associated with each method 
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The quantitative data generated by the needs assessment can be analyzed and reported by 
means of procedures generally used in conjunction with surveys Common descriptive and infer- 
ential statistics are also appropriate when the data are amenable to such treatment In general, 
graphical displays of the findings enhance comprehension Qualitative data are more meaningful 
when presented in narrative form, but this does net exclude the use of matrices and tables when 
appropriate. 

A needs assessnr nt report usually consists of both a technical report and a brief executive 
report. The technical report should describe all the procedures used and present all the data col- 
lected, data analyses, interpretations, findings, conclusions, and recommendations The executive 
report is a brief document, usually about two to three pages, that explains how the problem was 
identified, summarizes the needs in terms of the discrepancy between actual and desired states of 
affairs, and presents the causal analysis of the needs. 



Needs Assessment for Specified Purposes 

Participant-oriented employment assistance programs, such as those sponsored by the 
government under JTPA or by UAW-Ford in its Targeted Vocational Retraining, focus their needs 
assessment on participant needs and labor market needs. In the Marion Job Training Center in 
Marion, Ohio, for example, the ciiei.t needs assessment process consisted of a battery of tests to 
assess basic skills, general knowledge, achievement, aptitudes, interests, and attitudes. Individual 
counseling is also used to determine the needs, interests, and goals of participants. To assess 
labor market needs, the Center relies upon the projections and descriptions in the Occupational 
Outlook Handbook, a list of occupations in demand in the State, and local job listings. 

The UAW-Ford Reemployment Services Assistance Centers are designed to help clients 
determine their own training/career development needs. These centers screen proposals by pro- 
viders of training services and identify those with the greatest potential for meeting their clients' 
training and reemployment needs. Program selection criteria are based oon client needs. Schedul- 
ing, transportation, and location of the training program are considered, as well as prerev.uisite 
skills and competencies. Client interests and short- and long-term goals and the transferability of 
training are also examined. In addition, the UAW-Ford request for proposal process stipulated that 
recommended programs show a strong assurance of labor market demand and a commitment 
from local firms to hire graduates of the programs for jobs with minimum entrance level salaries. 

Needs assessment for customized training usually follows a somewhat different process. Post- 
secondary institutions often rely cn a company's own assessment of its needs as a starting point 
for program development. Occasionally, however, instructors may discover a different set of needs 
when the program is under way. Employees may lack some basic skills or demonstrate non- 
training-related communication problems. Some of these surprises might be averted through 
screening candidates for the skill training. A collaborative approach to needs assessment and pro- 
gram planning and design can be very helpful, as it enables the postsecondary institution to verify 
the identified training needs at several levels in the company before the program actually begins. 
An example of this extended program planning process is provided by Trident Technical College 
in table 2. 

The development of training programs directed at meeting emerging industry needs in specific 
skill areas relies heavily on the tack force approach. In the New York Finger Lakes area, regional 
task forces in five growing industries represent the needs of individual companies that are too 
small to demand customized training. Initially, industry representativas meet with college adminis- 
trators and training coordinators to discuss the local need for a training program. An employer 
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TABLE 2 

CUSTOMIZED ON-SITE INDUSTRIAL POSTEMPLOYMENT COURSE 


oxep 


TTP 


inuusxry 


Cooidinator 

Unit rc 

nuui 9 


Instructor 

Mm ire 


Secretarial 

H n i ire 

nuui 9 


A. Initial Meeting 
with Industry 
Representative 


Discuss TTC 

— Training Capabilities 

— Objective 

— Goals 


Discuss 

- Existing Training 

- Future Training Needs 

- Scheduling Alternatives 

- Target Audience 








B. Initial Course 
Planning 


Discuss 

- Course Material Available 

- Presentation Options 

- Material Organization 

- Train. Location Options 

- Support Requirements 


Provide 

— Technical Details 

— Content Agreement 

— Location Agreement 








C. Formal Course 
Planning 


Review 

- Technical Details 

- Course Proposal 

- Cost Analysis 

- Instructor Options 

- Textbook Options 


Approve 

- Course Proposal 

- Cost Proposal 

- Text 








D. Instructor 
Selection 


Acquire Resume of 
Fotentia' Instructors 

— Arr. y 3 Interviews 

- Prepare Temporary 
Employment Agreement 


— Review Resumes 

— Interview Instructors 

— Approve Instructor 
Selection 








E. Course Tsvelopment 


Prepare 

— Course Objectives 

— Performance Objectives 

- Course Syllabus 

- Tests 


Provide 

- Subject Matter Specialists 
(If Necessary) 

- Company Visitation 








F Course Approval 


— Review Complete Training 
Package 

— Provide Final Course Proposal 


- Approve Training Plan 

- Select/Noufy Students 








Cm 1 mnlpmpntatinn Inct 


Rpnin Traininn f^nntrart 

w cy ill i i ail ill i u v> kj 1 1 1 1 □ i 


Imolpment Traininn SchpHulp 

|l 1 1 pigl l IC l l I 1 lull III iy wwMCUUIC 








H. Monitoring & 
Evaluation 


— Critique Instructor 

— Identify & Correct Problems 

— Insure Support Materials 


— Supply Feedba:k 

— Determine Trail, ing 
Effectiveness 








1. Travel Hours & Cost 












J. Preparation of BMIing 












SOURCE: Trident Technical College, Charleiton, South Carolina. 
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survey may be used to help determine the level of demand and narrow down specific skills and 
competencies. D ,,r ing the needs assessment and program planning process, the task force may 
meet several times to define needs; refine program objectives, validate curriculum, equipment, and 
learning materials; end suggest or even approve instructors for the program. The task force per- 
forms much same functions as the advisory council utilized extensively in vocational educa- 
tion, but it is generally seen as a transitory body. If the program becomes an established certificate 
or even degree program, task force members are ideal candidates for the advisory council. Advi- 
sory committees that are already assembled for longer, established programs in related areas may 
be used to validate the need for and design of individual short courses. At Hennepin Technical 
Centers' south campus, in Eden Prairie, Minnesota, advisory committees oversee some 90 percent 
of all short programs. 

Needs assessment is an important activity that deserves considerable attention, as it is the 
starting point for efforts to design and improve training programs. If properly conducted, it can 
save time, effort, and money. 



Input Evaluation 

The main purpose of input evaluation is to provide the information needed to design a pro- 
gram to meet identified goals and objectives. More specifically, input evaluation assesses existing 
resources and determines additional resources needed to bring about desired changes (i.e., to 
attain the intended objectives). 

Anderson et al. (1977) indicated that input evaluation "describes the resources available and 
determines the best use of those resources in terms of cost and benefits, resulting in a design to 
meet the goals" (p. 118). Stufflebeam and Shinkfield (1985) argue that 

the overall intent of an input evaluation is to help the clients consider alternative pro- 
gram strategies in the context of their needs and environmental circumstances and to 
evclve a plan that will work for them; another important function is to help the client 
avoid the wasteful practice of pursuing proposed innovations that predictably would fail 
or at least waste resources, (p. 173) 

Anderson et al. (1977) note that "input evaluation results in a proposed design for the implementa- 
tion of a program" (p. 119) and raises a number of important issues: "feasibility of accomplishing 
goals, availability of strategies to meet goals, potential costs of various strategies, probability of 
success of various strategies based on past experiences, the utilization of staff, scheduling of activ- 
ities, and the need to call on outside resources" (p. 119). Most of these issues should be examined 
carefully during the design and planning of short-term skill training programs. 

There is no single approach to the conduct of an input evaluation. Some of the most common 
approaches in current use are the Program Evaluation and Review Technique (PERT), the De'ohi 
technique, and cost-effectiveness analysis (Anderson et al. 1977). Stufflebeam and Shinkfiei 
(1985) identify a set of procedures that can be used for conducting an input evaluation: 

• Review current practice, by examining the literature, visiting exemplary programs, obtain- 
ing consultant advice from government agencies, and inviting local staff to generate 
proposals. 

• Examine promising approaches in terms of potential effectiveness and feasibility. 
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• Identify the critet ta that the proposed program must neet and assess potential program 
designs according to these criteria and in terms of effectiveness and feasibility. 



• Gather feedback from staff and administrators about the feasibility of installing proposed 
programs within the constraints of available resources 

Stufflebeam and Shinkfield (1985) also identify an innovative approach for conducting input 
evaluation entitled "The Advocacy Team Technique." The process calls for the selection of two or 
more teams of experts. They are given the program objectives, specifications for designing a pro- 
gram proposal, and the criteria by which their input will be judged. The resulting proposals are 
rated by a panel of experts or pilot-tested against specific criteria. Local staff must then select the 
alternatives that best meet program needs. As the authors point out, this technique provides— 

• an explicit procedure for generating and assessing complete program strategies, 

• an explicit record of why a particular solution strategy was selected, 

• a forum that uses potential bias and competition in a constructive search for alternatives, 



• a means of involving personnel from the adoptin ; system. 

Two other useful sources of information for input evaluation are (1) the program review 
guidelines that outline standard? by which vocational education programs should be evaluated 
and (2) the program standards specifications that identify the space, equipment, facilities, and 
tools necessary for the delivery of specific vocational education programs. These two sets of docu- 
ments have considerable implications for the input evaluation and can be very useful. 

It is important to gather information from aP appropriate sources. The task force or advisory 
committee, employer, and/or internal curriculum development specialist can be particularly useful 
in providing evaluative input into the design of instructional programs. Advisory r». omittees and 
employers can validate program performance (task) objectives, as well as substantive content, 
materials, and equipment. Curriculum developer* an provide assurance that the design and 
sequencing of courses and curriculum material are appropriate to identified student needs and 
that they should facilitate learning. Figure 3 presents an example of an advisory committee 
assessment form utilized at District 916 Area Vocational Technical Institute. 

Criteri? that may be considered in designing skill training programs include the following 

• Efficient use of instructors and equipment without overcrowding 

• Adequate support services u \jet student needs (i.e., counselors, library or instructional 
materials center) 

• Qualified instructors 

• Sufficient time allotted to training 

• Homogeneous group of trainees who can be expected to learn at about the same pace 



and 
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CRITERIA FOR TASK LIST AND PROGRAM MATERIALS ASSESSMENT 




Proaram Name Proqram Number 






Date 






Advisory Council Committee 
Chair Signature 






Assistant Director 
Signature 




ITEM 


SAT. 


UNSAT. 


1. The program tasks appear to be valid (in other words, a new 
employee would be expected to perform these tasks). 






2. The content of the course material appears to be valid (in 
other words, the materials contain information that is 
needed for new employees to perform their jobs). 






3. Course material appears to be free from sexual, racial, 
and religious stereotyping. 






List any tasks or course material that you feel needs to be 
updated, deleted or added. Use the space below and the 
back of this sheet if needed. 



SOURCE District 916 Af9 Vocational Technical Institute, White Bear Lake, Minnesota 



Figure 3. Sample advisory committee assessment form 



0 
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Employers who are not experienced with training often underestimate the amount of time it takes 
Coordinators of customized training progra ns must be prepared to argue for the allowance of 
enough learning time for programs to be efficient. 



Process Evaluation 



Process evaluation is used to monitor the implementation of a training program Stufflebeam 
and Shinkfield (1985) identify four mam purposes of process evaluation 

• To provide feedback to managers and staff about the extent to which program activities 
are on schedule, are being carried out as planned, and are using the available resources 
efficiently 

• To provide guidance for modifying or explicating plans as needed 

• To assess periodically the extent to which program participants accept and are able to 
carry out their roles 

• To provide an extensive record of the program as it was actually implemented, including a 
comparison to what was intended, a full accounting of the costs incurred, and a record of 
how observers and participants judged the quality of the program 

Process evaluation, therefore, is essentially a formative type of evaluation, in which the focus 
is on program monitoring and improvement. Morris and Fitz-Gibbon (1978) suggest a four-step 
approach for conducting a formative evaluation: 

• boundaries of the evaluation. 

• Prepare a program statement 

• Monitor program implementation and the achievement of program objectives 

• Report to and confer with planners and staff. 

Due to the time constraints normally associated with short-term skill training programs, it is 
important to monitor program implementation to ensure that it follows the original plans. Any dis- 
crepancies should be identified and rectified immediately. Without close control, the original 
objectives can be missed and the entire training effort jeopardized. Process evaluation provides a 
quality control and quality assurance mechanism for short-term skill training. 

Quantitative input into process evaluation can be gained by monitoring attendance, perfor- 
mance on competency and skills tests, use of materials/supplies, and ongoing costs. Qualitative 
input may come from periodic observations of instruction by an administrator, department head, 
or training coordinator, and informal feedback from students. Greenville Technical Institute 
requires that deans or department heads assess classroom instruction and provide feedback to the 
instructor for any program longer than 2 weeks. In UAW-Ford programs, a client service adminis- 
trator monitors Targeted Vocational Retraining programs on a regular basis to assure that client 
needs are being met. The executive director of the Finger Lakes Regional Education Center for 
Economic Development in New York State maintains regular contact with training coordinators at 
the individual colleges and periodically visits ongoing programs to provide process evaluation. 
Monitoring both quantitative and qualitative data provides program administrators with a means of 
keeping the program "on target" and meeting the needs of participants. 
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Outcome Evaluation 



Outcome evaluation is part of the broad product evaluation process that assesses program 
outcomes and impact. Its mam purpose is to determine if the immediate program objectives have 
been met. Whereas the main focus of outcome evaluation is on immediate program effects, impact 
evaluation focuses on long-term effects. 

Stuffiebeam and Shinkfield (1985) suggest that outcome evaluation should determine both the 
intended and the unintended effects of a program. They also recommend that outcome evaluation 
should gather data from a broad range of constituents involved in a particular program. 

There are many levels of outcomes on which a program eva*uation can focus, such as individ- 
ual outcomes, group outcomes, service provider outcomes, and outcomes for service receivers 
and stakeholders. In most cases, the information needs of the client dictate the level upon which 
the evaluation effort should focus. In some instances it is more meaningful to examine the various 
levels of outcomes individually; in others, an aggregate of the*e outcome measures may better 
serve the decision-making process. 

Farley et al. (1985) state that outcome studies are similar to follow-up studies. Both are 
intended to identify short-term consequences and longer term impact resulting from vocational 
education programs. However, Farley et al. (1985) note that outcome studies differ from follow-up 
studies in that they are designed, by and large, "on the basis of systems analysis models in which 
program outcomes are related in some fashion to program inputs" (p. 33). 

Outcome studies of vocational education programs, whether they be short- or long-term, are 
used to provide information on the productivity, effectiveness, and accountability of the programs 
If, for example, a program placed 95 out of 100 of its participants in training-related jobs, one 
might conclude that the program was productive and effective. However, as Farley et al. note, cost- 
benefit or cost-effectiveness studies can provide a more sophisticated indicator of program 
accountability. 

There is no set of procedures specifically designed for conducting outcome evaluation, but a 
variety of approaches used for other purposes can be combined to generate the information 
needed. The following are some examples of methods for generating outcome data* 

• Pretest and posttest achievement 

• Follow-up studies of graduates 

• Follow-up studies of employers sponsoring training programs 

• On-the-job performance appraisals of graduates 

• Cost-effectiveness and cost-benefits analyses 

• Case studies on job sites 
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The National Center for Research in Vocational Education has developed documents that pro- 
vide very precise guidelines for conducting follow-up studies Following is a list of some of the 
more relevant studies: 



• Evaluating Student Satisfaction^ R Ponce and S J Franchak, 1981 

• Evaluating Employer Satisfaction, S J. Franchak, 1981. 

• Guidelines and Practices for Follow-up Studies of Former Vocational Education Students, 
S J. Franchak and J. E. Spirer, 1978. 

• Guidelines and Prt ctices for Follow-up Studies of Special Populations, S. J. Franchak and 
J. E. Spirer, 1979. 

The outcome measures obtained from outcome evaluations are compared to the original 
needs assessment in order to determine program success. The evaluation should diacuss any dis- 
crepancies between "what is" and "what should be." 

Stufflebeam and Shinkfield (1985) indicate that outcome evaluation is important for account- 
ability purposes. In some short-term skill training programs, such as JTPA programs, the funding 
agencies set specific program outcomes in terms of performance standards. JTPA was designed to 
be performance driven and to ensure the achievement of program objectives. Section 106 of the 
act pi'ts the responsibility on the J.S. Secretary of Labor and the State Governors to establish and 
apply specific performance standards to the programs. Following are the performance measures 
that must be recorded* 

Adults 

• Entered employment rate 

• Cost per entered employment 

• Average wage at placement 

• Welfare recipient entered employment rate 
Youth 

• Entered employment rate 

• Positive termination rate 

• Cost per positive termination 

Using these measures and guidelines established by the U.S. Secretary of Labor, the State 
Governors set standards for local service delivery areas, taking into consideration local factors that 
are likely to influence performance. As performance of JTPA programs depends to a large extent 
on environmental conditions, the standards must be set with these conditions in mind and may 
vary according to local conditions. It is also the Governors' responsibility to determine whether 
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these standards have been met and to provide the appropriate rewa r ds, sanctions, or technical 
assistance. The authors of the JTPA Performance Standard Project (CSR, Incorporated 1984) 
note 

Assessment of each service delivery area's performance is made difficult by the fact that 
programs operate under different conditions and the standards setting system attempts 
to account for conditions over which the program operator has little, if any, control A 
simple comparison of performance is neither appropriate nor equitable as programs are 
administered under different conditions in different localities, (p. 3) 

Table 3 provides a summary of the information that is typically required for assessing the perfor- 
mance of JTPA programs (CSR. Incorporated 1984). As table 3 shows, outcome evaluation can be 
very complex. With the growing concern for accountability in training, the traditional approach of 
evaluating training on the basis of participant reaction is not adequate. Participant reaction does 
not provide the valid and credible information required by funding agencies. Program providers 
can generate measure outcomes by more accurately triangulating information from various 
sources at various levels of validity and reliability. 

Finally, because short-term skill training is closely tied to employment outcomes, training is 
often evaluated only in terms of those employment outcomes and not on the basis of learning 
outcomes. If evaluation is to be useful, it must measure learning. It must indicate, for example, 
whether unemployed participants effectively learned job-seeking and preparation skills or whether 
their training was inappropriate or inadequate Positive results are not always due to the training 
program and should be examined. If the business cycle suddenly takes a turn for the better, it will 
be necessary for evaluators to be able to sort out the positive effects of training and the effects of 
the improving economy. 



Impact Evaluation 

Training evaluation often ends with the determination of whether the program objectives have 
been met and the identification of the immediate outcomes of training. However, other important 
short- and long-term effects should be assessed if the full impacts of training are to be known 
These impacts can be measured in terms of the broader effect or worth of training efforts for the 
individual, for the business community, and for society as a whole. 



Individual Impact 

For the individual participating in the training program, impact can be indicated in several 
ways: 

• Promotion (if the individual was already employed) 

• Increased job satisfaction 

• Integration or reintegration into the labor market after a period of unemployment 

• Higher earnings 

• Increased occupational mobility and career growth 
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INFORMATION REQUIRED BY GOVERNORS TO DETERMINE 
IF JTPA PROGRAMS HAVE MET PERFORMANCE STANDARDS 







Terminee 








Characteristics 




Planning Process 


Measures 


No. or Percentage 


Others 


Adults 








No. 'ho terminated 


Entered employment 


Female 


Averaoe no of weeks 


No. who entered 


rate 


45- to 54-year-olds 


of participation of 


employment 


No. terminated as a 


55 years and over 


all adult terminees 


Total expenditure 


percentage of total 


Blacks 




for adults 


no. terminated 


Hispanics 




Wage for all adults 


Cost per entered 


Other minor, lies 




who entered employ- 


employment 


Dropouts 




ment at termination 


Total expenditure 


Handicapped 






divided by no. who 


Ul claimants 






entered employment 


Welfare recipients 






Average wage placement 








Average wage for all 








adults who entered 








employment at the 








time of termination 








Welfare entered 








employment at ter- 








mination expressed 








as a percentage of 








no. of welfare 








recipients who 








terminated 






Youth 








No. who entered 


Entered employment 




Average no. of weeks 


employment at 


rate 




participation for 


termination 


No. who entered 




all youth 


No. of youth who had 


employment as a 




Local economic condi- 


a positive termina- 


percentage of no. 




tions for the SDA 


tion (both entered 


who terminated 




Unemployment rate 


employment and 


Positive termination 




Average wage for the 


employability 


No. having a positive 




area 


enhancement 


termination as a 






terminations) 


percentage of no. 






Total expenditure 


terminated 






for youth 


Cost per positive 








termination 








Total expenditure for 








youth divided by no. 








of youth having 








positive termination 
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• Enrollment in additional training or education 



Training impact on the individual level is usually assessed through follow-up studies con- 
ducted at various intervals to determine both short- and long-term impacts Longitudinal analyses 
based upon National databases sucf as the National Longitudinal Survey (NLS) provide impact 
data for various types of training anc these data may be useful in program design but are not use- 
ful in assessing the impacts of specific training programs. 



Organizational Impact 

Impact at the organizational level is usually more difficult for training institutions to document 
However, through collaboration with business and labor, evaluators can assess the worth of train- 
ing to business and industry In general, the most significant impact of training on the business 
community is increased productivity. 

Two dimensions must be considered in the assessment of productivity. The unit of analysis 
may be one of nine levels— individual, work group, department, function, plant, division, firm, 
industry, or nation. The measurement scope refers to the time interval used, which may be in min- 
utes, hours, days, weeks, months, and so on. These two dimensions can be placed on a matrix to 
help focus the measurement of any observable change in productivity. 

Other indicators of training impact at the organizational level are reduced accident rates, 
lower absenteeism, fewer rejects and less waste, higher morale, and improved quality of products. 
The training program may also have impact on the nature of organization and management, as 
was the case in one company visited in New York State. There the effect of the training program 
did not simply manifest itself in the employees* ability to understand and correct production prob- 
lems The implementation of training over a 9-month period resulted in major changes in the way 
line supervisors viewed their roles and the way they interacted with production line employees 



Societal Impact 

Whatever the original intent of a training program, the program will always have impact on 
society. In fact, some training programs such as JTPA are specifically designed to have direct 
societal impact. Examples of training impact on society are— 

• reduced unemployment; 

• equal employment opportunities to youth, the elderly, women, and minorities, 

• the return of welfare recipients to the labor market; 

• a more balanced standard of living among citizens; and 

• political stability in the country. 

Societal impact may not be a justifiable goal for all types of training and often does not fall 
within the competence and responsibilities of postsecondary institutions. However, institutions 
can assist in efforts to impact society positively by keeping accurate records of all their training 
activities. 
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The New York State Education Department (1984) has developed two interesting models for 
determining the impact of short-term skill training programs* the impact model for production and 
service organization, the impact model for specialized training situations These are discussed at 
length in their publication Training and Economic Development: The Impact of Industry-Specific 
Training on Business and the Economy of New York State. 

Knowing that a training program has been effective in meeting its intended objectives is 
important but not adequate if the results, the worth, or the short- and long-term impact are not 
known. Postsecondary institutions should endeavor to assess the impact of their training programs 
whenever possible. 



Short-term skill training is experiencing a boom. Substantial amount? are invested each year 
in short-term training by government agencies, schools, and private corporations. In spite of this 
massive deployment of resources, there has been little effort to assess training systematically in 
relation to financial investments. 

The process of assessing a training program on the basis of its benefits in relation to cost is 
referred to as cost-benefit analysis. Cost-effectiveness analysis similarly addresses effectiveness in 
relation to cost. Kim (1977) defines the two key terTS as follows: 

• Cost-benefit analysis involves the determination of the cost and the benefits of a single 
program for the purposes of improving »he rr.tio of benefits to cost. 

• Cost-effectiveness analysis is an analytical technique for assessing the outcomes of a 
program in relation to its specified objectives and against its cost, (p 3) 

The two are discussed together here because both deal with what might be called the return on 
investment in a program. 

Kim also indicates that vocational education programs can use cost-effectiveness/cost-benefit 
analysis as a tool in systems analysis evaluation and planning. Wentlmg (1980) identifies even 
more reasons for conducting cost-analysis evaluations in education: 

• To provide information for resource allocation decisions 

• To promote better utilization of facilities 

• To determine optimum staff assignments 

• To determine optimum scheduling or sequencing of courses 

• To determine the optimum number of participants for a particular program 

• To decrease the cost of high-cost/low-in^dence programs 

• To determine whether to finance the development of a new program 

• To compare alternate programs 

• To account for expenditures 



Cost-Benefit Analysis 



ERIC 




Kim has designed one of the most comprehensive models for analyzing cost-effectiveness/ 
cost-benefit in postsecondary vocational education programs. This model is based on four major 
components* social demands; social support, student input; and object. ^es, programs, and cost 
The cost-effectiveness/cost- benefit measures include student outputs, noneconomic benefits, and 
economic benefits. 

Conducting an Analysis 

If a cost-benefit analysis is to be successful, all the steps in the process must be identified and 
planned in advance. The steps identified by Kim provide excellent guidelines 

Planning 

• Determine the purposes and scope of the analysis 

• Identify appropriate resources. 

• Develop a study plan and time schedule. 
Implementing 

• Determine program objectives and target goals 

• Assess program outcomes. 

• Analyze direct or indirect program costs. 

• Compute cost-effectiveness/cost-benefit measures 



• Compute and interpret the measures and draw conclusions. 

• Utilize the results for program evaluation, development, and planning. 

Determining costs and benefits. The cost of a training program can be broken down into pro- 
gram development costs and program operation costs. Program development costs may include 
monies spent for: analysis, design, development, implementation, and evaluation and revision. 
Program operation costs include administrative costs, instructional staff costs, support staff costs, 
building and services costs, materials and supplies costs, insurance costs, and maintenance costs 
It is more difficult to assess with a reasonable degree of accuracy the benefits derived from train- 
ing investment, because often these benefits are intangible (Wentling 1980). 



Utilizing 
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CHAPTER 5 



CHOOSING AN EVALUATION APPROACH 



Because of the nature of short-term skill training programs and their du r ation, many of the 
approaches commonly used in the evaluation of long-term training programs may not provide the 
information needed. Short-term skill training programs are so numerous and diverse that system- 
atic evaluation of an individual program may not be feasible or desirable. The payoff in terms of 
the usefulness of the information may not justify the commitment in terms of financial and human 
resources. Furthermore, timeliness is such a critical factor in almost all short-term skill training. 
Courses may be so urgently needed that planning must proceed without the feedback generated 
from evaluation efforts. The results of evaluation may become available too late to influence any 
decision-making process. 

Many short-term skill training programs are one-time training efforts. In such cases evaluation 
results canno* be used to revise and improve programs, and the function o' evaluation becomes 
blurred. Moreover, when an institution or employer offers many short-terrr programs, it may not be 
possible to use the information from evaluations of every program. If each program is systemati- 
cally evaluated, already busy decision makers may be flooded with information they cannot pro- 
cess and act upon. 

These difficulties call for alternatives to the strategies traditionally used for assessing long- 
term training programs. This chapter discusses two promising techniques for evaluation of short- 
term skill training programs, aggregate program review and peer review, and concludes with the 
postcard model— an abbreviated but often useful method of evaluating very short programs. 

Aggregate program review and peer review are not innovative in themselves but are models of 
evaluations that have been used elsewhere. However, the idea of using them for short-term skill 
training assessment is innovative. These models hold the potential for enriching the evaluation of 
short-term programs. 



Program review is an approach to program evaluation that has been and is still being used by 
many universities for evaluating their academic programs and by many State boards of education 
for assessing their vocational education programs. 

For these users, program review generally focuses on the evaluation of the overall effective- 
ness of all long-term program offerings at the institutional level. For example, Ohio's Program 
Review for Improvement, Development, and Expansion (PRIDE) is a comprehensive evaluation 
system designed to assist vocational educators in planning and implementing quality vocational 
education programs (State of Ohio 1984). 



Aggregate Program Review 
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The State of Arkansas has a similar program operation. The purpose of the Arkansas \ na- 
tional Evaluation System is "to provide a vehicle for the systematic collection of information for 
use in planning, reporting, and decision making to be used for the improvement of vocational edu- 
cation programs" (Arkansas Department of Vocational and Technical Education, n.o.; The follow- 
ing are the objectives: 

• To promote and assist local education agencies and postseconoary vocational institutions 
in providing high quality vocational education programs 

• To foster maximum utilization and accouruability of State and Federal funds allocated for 
vocational education programs and activities 

• To provide necessary information for planning, reporting, and decision making at the 
State and local levels (p. 10) 

Many other States have similar program review systems wi f * similar objectives. The essential 
difference between proqram evaluation and aggregate program review is that the former focuses 
on the evaluation of individual programs and the latter is geared toward the asses r /nent of a range 
or set of programs. Most State program reviews consist of two major components: a self- 
assessment conducted by local staff and a visitation by an outside team. 

Self tsessment. Self-assessment involves alt the local staff members of the training institution 
in a joint evaluation effort. An organizational structure proposed by the Arkansas Department of 
Education has proven effective for the self-assessment process. This structure includes a steering 
committee and a series of assessment subcommittees, each assigned to an individual program. 
The steering committee ?"d one administrator or supervisor usually have the responsibility for 
appointing and providing leadership for the totp.i effort. The assessment subcommittees conduct 
the evaluation procedures and report to the steering committee; all the findings of all these reports 
are compiled into a summary report. 

Site visitation. In the Arkansas system site visitation is performed by a team of external evalu- 
ators selected by the director of the vocational and technical education division of the State 
department of education upon the recommendation of local supervisors or administrators. The 
team usually spends 1 day at a site. Larger schools may r-iuire 2 days. The leader of the visiting 
team assigns team members to subgroups that are responsible for specific evaluation tasks. Each 
subgroup completes the assigned tasks according to specific guidelines and criteria, and after the 
r.ie visit each subgroup drafts an evaluation report. It is then the team leader's responsibility to 
complete the final evaluation report that summarizes the findings and submit <* to the institution 
that has been evaluated. The school must then respond to this report, indicating the actions it will 
take to meet the recommendations of the site visit evaluators. This response becomes the school's 
Vocational Plan of Improvement. 

A sin lar procedure could be uswd 'or the over ill evaluation of short-term skill trainir g pro- 
grams a* e secondary and postsecondary levels. Such an approach should he'p to overcome 
some of the difficulties usually encountered in evaluating individual short-term programs. 

The panel of experts convened for the project on which this guidebook is based suggested 
monitoring a limited number of specific characteristics routinely as the baseline process. Program 
costs, enrollments, and placement rates are examples of characteristics that might be monitored 
for each program. For institutions with computer-managed information systems, monitoring of 
multiple programs in this way requires les f . iffort than in the past and can be a cost-effect' e 
mechanism for conducting aggregate program review. 
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As the database or- various programs accumulates, benchmarks for evaluation can be estab- 
lished and a procedure for flagging exceptional programs developed These exceptional programs 
can then be evaluated r ~ efully to determine the re" ^ for their success (or failure) Periodic 
assessment of randomly selected programs may inducted to validate monitoring criteria and 
flagging standards 

At Trident Technical College, each type of short-term training program is overset n by a Unit 
Coordinator. These coordinators meet with the Dean of External Programs once a month to moni- 
tor all programs. They review a number of benchmarks: 

• Current program activities 

• New contracts 

• Expenditures and revenues per program 

• Number of companies served 

• Number of contact hours 

• Comparison of ongoing programs with n<?w ones 

In this way they are able to assess programs and avert potential p r oblems by shifting emphasis 
between units as appropriate. 



The second strategy proposed by the panel of experts is peer review. Peer revhw evaluation is 
essentially an evaluation effort conducted by knowledgeable individuals from outside the institu- 
tion Like aggregate program review, peer review focuses on the assessment of the overall pro- 
gram offerings. This process can be self-initiated or commissioned by some State authorities. 

The purpose of peer review is to provide an objective assessment of the training program 
Because external peers have no stake in the evaluation results, the results should be objective and 
therefore more meaningful. 

If peer review evaluation is to be used, the identification and selection of appropriate peers is 
of major importance. These reviews can be selected from similar institutions within the same State, 
from similar institutions outside the State, or from the ranks of private consultants with expertise in 
training and development. The use of peers from similar institutions within the same State is per- 
haps the most aconomical alternative. Depending upon the agreement reached, there can be fees 
attached for 'he service or an exchange of services between institutions. Arrangements with 
reviewers froi.i other States can be patterned in the same way, but additional travel costs are 
involved. Private consultants can perhaps provide the most specialized and expert services in the 
evaluation— and the most costly. Bear in mind that peer selection is an important issuo with poten- 
tially important economic and political consequences. Peer-to-peei evaluation by experts from 
other States may require the review of sensitive issues that State legislators or school administra- 
tors do not want to share with outsiders. 



Peer Review 
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Peer review evaluation strategies can focus on various aspects of skill training programs, from 
~^eds assessment to program design and implementation. Peer evaluation of program outcomes 
and impacts is not appropriate because of the time and resources usually required for these pro- 
cesses. However, peer reviewers can be used to audit internal evaluations of program outcomes 
and impact conducted by in-house staff. When reviewers are carefully chosen and their agenda 
well thought out, the peer review technique can greatly help to alleviate the problems often 
encountered in evaluation 



Using Evaluation Models 

For short-term training courses that are brief and run only once, extensive evaluation may not 
be appropriate. At the same time, some form of evaluation is always desirable. Aggregate program 
review and peer review normally generate a great deal of useful information for the development of 
future programs and the improvement of existing programs. For established programs, other evalu- 
ation approaches may be beneficial. 

In modifying an evaluation approach, care should always be taken; the benefits of the 
approach depend on following procedures with some accuracy. In peer review, for example, the 
evaluation hinges on the good judgement and expertise of the reviewers. In aggregate program 
review, the orderly acministration of the self-assessment is crucial. 

For training seminars that last no more than two days, an abbreviated form of evaluation can 
be usefLl. The panel of experts designed a minimal evaluation approach that can be useful for 
short seminars. This postcard model uses a little from three evaluation approaches. It involves the 
following steps: 

• Needs assessment 

• Outcome measure 

• Follow-up 

The needs assessment may be minimal if the company/client has already determined what the 
training need is. That is, a company might decide that a business writing course is needed. The 
needs assessment on the part of the educational provider can then focus on the questions, What 
arc* the specific needs for business writing for that particular organization? end At which points are 
employees deficient? 

An outcome measure may be as simple as a "smile sheet" or "happiness index" at the end of 
the course (i.e., a student reaction questionnaire). If the client company and the provider have 
agreed in advance that this type of self- reported measure is an appropriate indicator of success, 
then a simple, short questionnaire may be all that is required for this aspect of the evaluation 
(Clients should not, however, be er.<:ouraged to see self-report as a very useful way to measure 
outcomes.) 

Some form of simple follow-up is needed to ascertain whether and to what extent participants 
have used the skills and knowledge that they learned and whether employers are satisfied with the 
training the learner employee received. This follow-up may be in the form of a postcard sent to 
either participants or their direct supervisors, requesting tr,em to mark appropriate responses. 
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The postcard model is recommended cautiously. Its real value is probably to illustrate how 
important methods of evaluation, such as needs assessment, can be worked into even the briefest 
training program. The intention of this publication has been to indicate the many forms evaluation 
can take and the range of information it can generate. Those who are interested in exploring an 
approach mentioned here can refer to the literature discussing the model that is cited in the text 
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APPENDIX A 



SITE VISIT LOCATIONS 



Columbus Technical Institute 
Columbus, Ohio 

District 916 Area Vocational and Technical 

Institute 
White Bear Lake, Minnesota 

Finger Lakes Regional Educational Center 

for Economic Development 
Mount Morris, New York 

G C.A/Tropel, Inc. 
Fairport, New York 

Greenville Technical College 
Greenville, South Carolina 

Hennepin Technical Centers 
Intermediate District 287 
Plymouth, Minnesota 

Private Industry Council, Inc. 
Crawford County Job Training Center 
Bucyrus, Ohio 

Private industry Council, Inc 
Marion Job Training Center 
Marion, Ohio 



Trident Technical College 
Charleston, South Carolina 

Schlegal Corporation 
Rochester, New York 

UAW-Fora National Development and 

Training Center 
Dearborn, Michigan 

Zone 1: UAW-Ford Career Services and 

Reemployment Assistance Center 
Macomb Community College, site 
Fraser, Michigan 

Zone 2. UAW-Ford Career Services and 

Reemployment Assistance Center 
CareerWorks, Inc., site 
Detroit, Michigan 

Zone 3: UAW-Ford Career Services and 

Reemployment Assistance Center 
Jewish Vocational Services, site 
Dearborn Heights, Michigan 
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APPENDIX B 



TECHNICAL ADVISORY PANEL MEMBERS 



ANNBLALOCK 

Special Assistant to the Deputy Commissioner 
State of Washington, Employment Security Dept 
Seattle, Washington 

OLIVER CUMMINGS 
Manager, Evaluation Services 
Arthur Anderson and Company 
St. Charles, Illinois 

WINIFRED DELOYAZA 
Executive Director 

Interorganizational Relationships Ltd. 
Altamont, New York 

RUSSELL PAULSEN 

Vice President for Development 

North Central Technical Institute, and 

Executive Director, National Council for Occupational Education 
Wausau, Wisconsin 
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